What is the purpose of rayon::iter::IntoParallelIterator::into_par_iter for converting collections into parallel iterators?
into_par_iter converts a collection into a parallel iterator that consumes the collection and distributes its elements across multiple threads for parallel processing. Unlike par_iter which borrows the collection, into_par_iter takes ownership, allowing the parallel iterator to own the elements directly. This enables more efficient parallel operations when you no longer need the original collection after iteration, as it avoids reference counting overhead and allows transformations that consume elements.
Understanding into_par_iter Basics
use rayon::prelude::*;
fn main() {
let data = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
// into_par_iter takes ownership of the vector
let sum: i32 = data.into_par_iter()
.sum();
println!("Sum: {}", sum);
// data is no longer accessible here - it was consumed
// println!("{:?}", data); // Error: value borrowed after move
}into_par_iter consumes the collection and owns elements during parallel processing.
par_iter vs into_par_iter
use rayon::prelude::*;
fn main() {
let data = vec![1, 2, 3, 4, 5];
// par_iter borrows the collection
// Returns parallel iterator over references (&i32)
let sum_refs: i32 = data.par_iter()
.map(|x| *x * 2)
.sum();
println!("Sum with refs: {}", sum_refs);
println!("Data still available: {:?}", data); // data is still valid
// into_par_iter takes ownership
// Returns parallel iterator over owned values (i32)
let sum_owned: i32 = data.into_par_iter()
.map(|x| x * 2)
.sum();
println!("Sum with owned: {}", sum_owned);
// data is consumed - no longer accessible
}par_iter borrows; into_par_iter takes ownership.
Ownership Enables Efficient Transformations
use rayon::prelude::*;
fn main() {
// When you own elements, you can transform them in place
let data = vec!["hello".to_string(), "world".to_string(), "rust".to_string()];
// With par_iter, you'd have to clone or create new Strings
let uppercase_borrowed: Vec<String> = data.par_iter()
.map(|s| s.to_uppercase()) // Creates new Strings
.collect();
println!("Original still exists: {:?}", data);
// With into_par_iter, you can reuse allocations
let data = vec!["hello".to_string(), "world".to_string(), "rust".to_string()];
let uppercase_owned: Vec<String> = data.into_par_iter()
.map(|mut s| {
s.make_ascii_uppercase(); // Transform in place
s
})
.collect();
println!("Uppercase: {:?}", uppercase_owned);
// No intermediate clones needed
}Ownership allows in-place transformations without cloning.
Converting Between Collection Types
use rayon::prelude::*;
use std::collections::HashSet;
fn main() {
let data = vec![1, 2, 3, 4, 5, 1, 2, 3];
// Convert Vec to HashSet in parallel
let set: HashSet<i32> = data.into_par_iter()
.collect();
println!("Unique elements: {:?}", set);
// Convert back to Vec with transformation
let doubled: Vec<i32> = set.into_par_iter()
.map(|x| x * 2)
.collect();
println!("Doubled: {:?}", doubled);
}into_par_iter is natural when converting between collection types.
Working with Owned Data
use rayon::prelude::*;
struct ProcessedItem {
id: usize,
data: String,
computed: u64,
}
fn process_item(id: usize, data: String) -> ProcessedItem {
// Expensive computation
let computed = data.len() as u64 * id as u64;
ProcessedItem { id, data, computed }
}
fn main() {
let items: Vec<(usize, String)> = (0..1000)
.map(|i| (i, format!("item-{}", i)))
.collect();
// into_par_iter lets us consume the tuples
let processed: Vec<ProcessedItem> = items.into_par_iter()
.map(|(id, data)| process_item(id, data))
.collect();
println!("Processed {} items", processed.len());
// The String data is moved into ProcessedItem
// No clones needed because we owned the data
}Owned parallel iteration enables efficient data transformation pipelines.
Memory Efficiency
use rayon::prelude::*;
fn main() {
// Large collection where we want to avoid clones
let large_data: Vec<Vec<u8>> = (0..100_000)
.map(|i| vec![i as u8; 100])
.collect();
// With par_iter, each thread holds references
// Reference overhead for large numbers of items
let sizes_borrowed: Vec<usize> = large_data.par_iter()
.map(|v| v.len())
.collect();
// With into_par_iter, we own the data
// Can transform without keeping original
let transformed: Vec<Vec<u8>> = large_data.into_par_iter()
.map(|mut v| {
v.push(255); // Modify in place
v
})
.collect();
// large_data is gone - memory was reused efficiently
println!("Transformed {} items", transformed.len());
}into_par_iter enables memory-efficient transformations on large collections.
Collecting Results
use rayon::prelude::*;
fn main() {
// into_par_iter supports various collection targets
let data = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
// Collect into Vec
let doubled: Vec<i32> = data.clone().into_par_iter()
.map(|x| x * 2)
.collect();
// Collect into HashSet (parallel deduplication)
let unique: std::collections::HashSet<i32> = data.clone().into_par_iter()
.collect();
// Collect into BTreeMap
let mapped: std::collections::BTreeMap<i32, i32> = data.clone().into_par_iter()
.map(|x| (x, x * x))
.collect();
// Collect with custom type
let summed: i32 = data.clone().into_par_iter()
.sum();
println!("Doubled: {:?}", doubled);
println!("Unique count: {}", unique.len());
println!("Mapped: {:?}", mapped);
println!("Sum: {}", summed);
}into_par_iter results can be collected into any FromParallelIterator type.
Real-World Example: Data Pipeline
use rayon::prelude::*;
use std::collections::HashMap;
#[derive(Debug, Clone)]
struct Record {
id: u64,
category: String,
value: f64,
}
fn main() {
let records: Vec<Record> = (0..10_000)
.map(|i| Record {
id: i,
category: format!("cat-{}", i % 10),
value: i as f64 * 0.1,
})
.collect();
// Process records in parallel, computing category statistics
let category_sums: HashMap<String, f64> = records.into_par_iter()
.fold(
|| HashMap::new(),
|mut acc, record| {
*acc.entry(record.category.clone()).or_insert(0.0) += record.value;
acc
}
)
.reduce(
|| HashMap::new(),
|mut a, b| {
for (k, v) in b {
*a.entry(k).or_insert(0.0) += v;
}
a
}
);
for (cat, sum) in category_sums {
println!("{}: {}", cat, sum);
}
}Complex parallel reductions work naturally with owned data.
Chaining Operations
use rayon::prelude::*;
fn main() {
let data = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
// Chain multiple operations with owned data
let result: Vec<i32> = data.into_par_iter()
.filter(|x| x % 2 == 0) // Keep evens
.map(|x| x * x) // Square them
.filter(|x| *x < 50) // Keep squares < 50
.collect();
println!("Result: {:?}", result); // [4, 16, 36]
// Operations that benefit from ownership
let strings: Vec<String> = vec!["hello".into(), "world".into(), "rust".into()];
let processed: Vec<String> = strings.into_par_iter()
.map(|mut s| {
s.push_str("_processed");
s
})
.collect();
println!("Processed: {:?}", processed);
}Owned parallel iterators chain efficiently with transformations.
Comparison with Sequential Iteration
use rayon::prelude::*;
fn main() {
let data: Vec<i32> = (0..1_000_000).collect();
// Sequential: into_iter takes ownership
let seq_sum: i32 = data.clone().into_iter()
.map(|x| x * 2)
.filter(|x| x % 4 == 0)
.sum();
// Parallel: into_par_iter takes ownership
// Distributes work across available CPUs
let par_sum: i32 = data.into_par_iter()
.map(|x| x * 2)
.filter(|x| x % 4 == 0)
.sum();
assert_eq!(seq_sum, par_sum);
println!("Both computed {}", seq_sum);
}into_par_iter is the parallel analogue of into_iter.
Flatten and Flat Map
use rayon::prelude::*;
fn main() {
let nested: Vec<Vec<i32>> = vec![
vec![1, 2, 3],
vec![4, 5, 6],
vec![7, 8, 9],
];
// Flatten nested collections
let flat: Vec<i32> = nested.into_par_iter()
.flatten()
.collect();
println!("Flattened: {:?}", flat);
// Flat map with transformation
let nested_strs: Vec<Vec<String>> = vec![
vec!["a".into(), "b".into()],
vec!["c".into(), "d".into()],
];
let upper: Vec<String> = nested_strs.into_par_iter()
.flat_map(|v| v.into_par_iter().map(|s| s.to_uppercase()))
.collect();
println!("Upper: {:?}", upper);
}into_par_iter composes with flatten and flat_map.
Working with Option and Result
use rayon::prelude::*;
fn main() {
let data: Vec<Option<i32>> = vec![Some(1), None, Some(3), Some(4), None, Some(6)];
// Filter and flatten options
let values: Vec<i32> = data.into_par_iter()
.flatten()
.collect();
println!("Values: {:?}", values); // [1, 3, 4, 6]
// Handle results
let results: Vec<Result<i32, &'static str>> = vec![
Ok(1),
Err("failed"),
Ok(3),
Err("error"),
Ok(5),
];
// Collect only Ok values
let ok_values: Vec<i32> = results.into_par_iter()
.filter_map(|r| r.ok())
.collect();
println!("Ok values: {:?}", ok_values); // [1, 3, 5]
}Owned iteration works naturally with optional and result types.
Performance Considerations
use rayon::prelude::*;
fn main() {
// When to use into_par_iter vs par_iter:
// Use into_par_iter when:
// 1. You don't need the original collection after
// 2. You want to transform elements in place
// 3. You're converting between collection types
// 4. Elements are expensive to clone
// Use par_iter when:
// 1. You need to use the collection again
// 2. You're only reading elements
// 3. Elements are cheap to reference
let data = vec![String::from("hello"); 1000];
// Expensive clones with par_iter
let _uppercase_borrowed: Vec<String> = data.par_iter()
.map(|s| s.to_uppercase()) // Clones each string
.collect();
// No clones needed with into_par_iter
let _uppercase_owned: Vec<String> = data.into_par_iter()
.map(|mut s| {
s.make_ascii_uppercase(); // Modifies in place
s
})
.collect();
// Trade-off: into_par_iter consumes the collection
// If you need the original, par_iter with clones might be preferable
}Choose based on whether you need the original and transformation cost.
Implementing IntoParallelIterator
use rayon::prelude::*;
use rayon::iter::{IntoParallelIterator, ParallelIterator};
// Custom collection that supports parallel iteration
struct MyCollection<T> {
items: Vec<T>,
}
impl<T: Send> IntoParallelIterator for MyCollection<T> {
type Iter = rayon::vec::IntoIter<T>;
type Item = T;
fn into_par_iter(self) -> Self::Iter {
self.items.into_par_iter()
}
}
fn main() {
let collection = MyCollection {
items: vec![1, 2, 3, 4, 5],
};
// Now we can use into_par_iter on our custom type
let sum: i32 = collection.into_par_iter()
.sum();
println!("Sum: {}", sum);
}Implement IntoParallelIterator for custom collection types.
Synthesis
Quick reference:
use rayon::prelude::*;
// into_par_iter takes ownership of the collection
let data = vec![1, 2, 3, 4, 5];
// Consumes vec, returns ParallelIterator over owned i32
let sum: i32 = data.into_par_iter().sum();
// Compare with par_iter which borrows:
let data = vec![1, 2, 3, 4, 5];
let sum: i32 = data.par_iter().sum(); // Iterates over &i32
// data is still valid here
// Key differences:
// par_iter -> borrows, yields references (&T)
// into_par_iter -> takes ownership, yields owned values (T)
// When to use into_par_iter:
// 1. Collection is no longer needed after iteration
// 2. Transforming elements (no clones needed)
// 3. Converting between collection types
// 4. Elements are expensive to clone
// 5. In-place modifications
// When par_iter is better:
// 1. Need to reuse collection after
// 2. Read-only access
// 3. Elements are cheap to borrow
// Common patterns:
// Transform in place
strings.into_par_iter()
.map(|mut s| { s.make_ascii_uppercase(); s })
.collect()
// Filter and collect
data.into_par_iter()
.filter(|x| x > 0)
.collect()
// Convert collection type
vec.into_par_iter().collect::<HashSet<_>>()
// Chain operations
data.into_par_iter()
.map(|x| x * 2)
.filter(|x| x < 100)
.collect()Key insight: into_par_iter is the ownership-taking counterpart to par_iter—it converts a collection into a parallel iterator that owns its elements. This matters for two reasons: first, it enables in-place transformations without the overhead of cloning or creating new allocations; second, it eliminates the reference indirection that comes with borrowed iteration. Use into_par_iter when you're consuming a collection as part of a parallel pipeline, especially when elements are expensive to clone or when you want to transform elements in place. Use par_iter when you need the collection afterward or when you're only reading data. The choice parallels the sequential distinction between into_iter (owned) and iter (borrowed).