Loading pageā¦
Rust walkthroughs
Loading pageā¦
rayon::iter::into_par_iter and par_iter for owned vs borrowed parallel iteration?Rayon's parallel iteration API provides two primary entry points: into_par_iter() consumes the collection and yields owned items, while par_iter() borrows the collection and yields references. The fundamental trade-off is ownership: into_par_iter() enables transformations that consume items (like .map() that returns new owned values) and allows the compiler to optimize based on exclusive ownership, but it destroys the original collection. par_iter() preserves the collection, allowing multiple parallel iterations or reuse, but constrains transformations to operate on references. This distinction mirrors the difference between into_iter() and iter() in standard Rust, extended to parallel execution with the same ownership semantics.
use rayon::prelude::*;
fn basic_parallel() {
let data = vec![1, 2, 3, 4, 5];
// par_iter: borrows, yields &i32
let sum: i32 = data.par_iter().sum();
println!("Sum: {}", sum);
// into_par_iter: consumes, yields i32
let product: i32 = data.into_par_iter().product();
// data is now consumed, can't use it again
}The key difference is whether the collection is borrowed or consumed.
use rayon::prelude::*;
fn ownership_demonstration() {
let data = vec![10, 20, 30];
// par_iter borrows the vector
let sum1: i32 = data.par_iter().sum();
let sum2: i32 = data.par_iter().sum(); // Can use again
println!("Sums: {} {}", sum1, sum2);
// into_par_iter consumes the vector
let owned: Vec<i32> = data.into_par_iter().collect();
// data is moved, can't use it anymore
// owned is now available for further use
let doubled: Vec<i32> = owned.into_par_iter().map(|x| x * 2).collect();
}par_iter allows multiple passes; into_par_iter enables single-pass transformations.
use rayon::prelude::*;
fn item_types() {
let data = vec![String::from("hello"), String::from("world")];
// par_iter yields &String
data.par_iter().for_each(|s: &String| {
println!("{}", s); // Can read, but not take ownership
});
// into_par_iter yields String (owned)
data.into_par_iter().for_each(|s: String| {
let _owned = s; // Can take ownership, move, transform
});
}
fn item_type_inference() {
let nums = vec![1, 2, 3];
// par_iter: item is &i32
nums.par_iter().for_each(|item| {
let _ref: &i32 = item;
});
// into_par_iter: item is i32
nums.into_par_iter().for_each(|item| {
let _owned: i32 = item;
});
}The item type differs: references with par_iter, owned with into_par_iter.
use rayon::prelude::*;
fn into_par_iter_use_cases() {
// 1. Transforming and consuming the collection
let data = vec![1, 2, 3, 4, 5];
let squared: Vec<i32> = data.into_par_iter().map(|x| x * x).collect();
// 2. Filtering and collecting into new structure
let data = vec![1, 2, 3, 4, 5, 6];
let evens: Vec<i32> = data.into_par_iter().filter(|&x| x % 2 == 0).collect();
// 3. Complex transformations that consume items
let strings = vec!["hello", "world", "rust"];
let lengths: Vec<usize> = strings
.into_par_iter()
.map(|s| s.len())
.collect();
// 4. When you don't need the original collection
let data = vec![vec![1, 2], vec![3, 4], vec![5, 6]];
let flattened: Vec<i32> = data.into_par_iter().flatten().collect();
// 5. Cloning is not needed - we already own
let owned_data: Vec<String> = vec!["a".to_string(), "b".to_string()];
let uppercased: Vec<String> = owned_data
.into_par_iter()
.map(|s| s.to_uppercase())
.collect();
}Use into_par_iter when you're transforming data and don't need the original.
use rayon::prelude::*;
fn par_iter_use_cases() {
let data = vec![1, 2, 3, 4, 5];
// 1. Multiple passes over same data
let sum: i32 = data.par_iter().sum();
let max: i32 = data.par_iter().max().unwrap();
let count: usize = data.par_iter().count();
println!("Sum: {}, Max: {}, Count: {}", sum, max, count);
// 2. Read-only operations
data.par_iter().for_each(|&x| {
println!("{}", x); // Just reading
});
// 3. Need to preserve collection for later use
let data = vec![String::from("hello"), String::from("world")];
let total_len: usize = data.par_iter().map(|s| s.len()).sum();
// data is still usable
assert_eq!(data.len(), 2);
// 4. Interacting with other borrowed data
let factors = vec![2, 3, 4];
let products: Vec<i32> = data
.par_iter()
.zip(factors.par_iter())
.map(|(&d, &f)| d * f)
.collect();
}Use par_iter when you need to preserve the collection or perform read-only operations.
use rayon::prelude::*;
fn performance_considerations() {
let large_data: Vec<String> = (0..100_000)
.map(|i| format!("item_{}", i))
.collect();
// par_iter with cloned() - clones every item
let cloned_upper: Vec<String> = large_data
.par_iter()
.cloned()
.map(|s| s.to_uppercase())
.collect();
// into_par_iter - no cloning needed
let owned_upper: Vec<String> = large_data
.into_par_iter()
.map(|s| s.to_uppercase())
.collect();
// owned_upper uses same memory as large_data did
// For Copy types, difference is minimal
let numbers: Vec<i32> = (0..100_000).collect();
// Both are efficient for Copy types
let sum1: i32 = numbers.par_iter().sum();
let sum2: i32 = numbers.clone().into_par_iter().sum();
}into_par_iter avoids cloning overhead for non-Copy types.
use rayon::prelude::*;
fn memory_optimization() {
// into_par_iter allows better memory layout
let data: Vec<String> = (0..10_000)
.map(|i| format!("item_{}", i))
.collect();
// par_iter requires cloning strings
let upper_from_borrowed: Vec<String> = data
.par_iter()
.map(|s| s.to_uppercase()) // Creates new Strings
.collect();
// into_par_iter can potentially reuse allocations
let upper_from_owned: Vec<String> = data
.into_par_iter()
.map(|s| s.to_uppercase()) // Consumes s, may reuse memory
.collect();
// The key insight: into_par_iter gives exclusive ownership
// allowing the compiler and runtime more optimization freedom
}Owned iteration allows more aggressive optimization by the compiler.
use rayon::prelude::*;
use std::sync::Arc;
#[derive(Debug, Clone)]
struct User {
name: String,
emails: Vec<String>,
}
fn complex_ownership() {
let users: Vec<User> = vec![
User { name: "Alice".into(), emails: vec!["alice@example.com".into()] },
User { name: "Bob".into(), emails: vec!["bob@example.com".into()] },
];
// par_iter: need to clone if we want owned output
let names_from_borrowed: Vec<String> = users
.par_iter()
.map(|u| u.name.clone()) // Must clone
.collect();
// into_par_iter: no clone needed
let names_from_owned: Vec<String> = users
.into_par_iter()
.map(|u| u.name) // Can extract without cloning
.collect();
// Arc allows sharing without cloning data
let shared_users: Vec<Arc<User>> = users
.into_par_iter()
.map(Arc::new)
.collect();
}With into_par_iter, you can extract fields without cloning.
use rayon::prelude::*;
fn mutable_iteration() {
let mut data = vec![1, 2, 3, 4, 5];
// par_iter_mut: borrows mutably, yields &mut i32
data.par_iter_mut().for_each(|x| {
*x *= 2;
});
assert_eq!(data, vec![2, 4, 6, 8, 10]);
// into_par_iter_mut doesn't exist
// Use into_par_iter if you want to consume and transform
let more_data = vec![1, 2, 3, 4, 5];
let doubled: Vec<i32> = more_data
.into_par_iter()
.map(|x| x * 2)
.collect();
}par_iter_mut provides mutable references without consuming the collection.
use rayon::prelude::*;
fn in_place_updates() {
let mut numbers = vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
// Modify in place
numbers.par_iter_mut().for_each(|n| {
*n = n * n;
});
assert_eq!(numbers, vec![1, 4, 9, 16, 25, 36, 49, 64, 81, 100]);
// Can do multiple passes
numbers.par_iter_mut().for_each(|n| {
*n += 1;
});
// Collection still usable
println!("Modified: {:?}", numbers);
}
fn combined_operations() {
let mut data = vec![String::new(); 1000];
// Initialize in parallel
data.par_iter_mut().enumerate().for_each(|(i, s)| {
*s = format!("item_{}", i);
});
// Process in parallel (read)
let total_len: usize = data.par_iter().map(|s| s.len()).sum();
// Modify in parallel again
data.par_iter_mut().for_each(|s| {
s.push_str("_processed");
});
}par_iter_mut enables efficient in-place parallel modifications.
use rayon::prelude::*;
fn cloning_comparison() {
let data = vec!["hello".to_string(), "world".to_string()];
// Option 1: par_iter + cloned
let upper1: Vec<String> = data
.par_iter()
.cloned() // Clones each string
.map(|s| s.to_uppercase())
.collect();
// data still valid
// Option 2: par_iter + clone in map
let upper2: Vec<String> = data
.par_iter()
.map(|s| s.clone().to_uppercase()) // Clone in map
.collect();
// Option 3: into_par_iter (no clone)
let upper3: Vec<String> = data
.clone()
.into_par_iter()
.map(|s| s.to_uppercase())
.collect();
// Option 4: into_par_iter (consumes)
let upper4: Vec<String> = data
.into_par_iter()
.map(|s| s.to_uppercase())
.collect();
// data no longer valid
}Choosing between borrowing and consuming depends on whether you need the original.
use rayon::prelude::*;
fn reference_operations() {
let data = vec![vec![1, 2, 3], vec![4, 5, 6], vec![7, 8, 9]];
// par_iter: nested iteration requires handling references
let sum_of_sums: i32 = data
.par_iter()
.map(|inner| inner.iter().sum::<i32>())
.sum();
// into_par_iter: owned inner vectors
let flattened: Vec<i32> = data
.into_par_iter()
.flatten()
.collect();
// Combining with other borrowed data
let coefficients = vec![2, 3, 4];
let scaled: Vec<i32> = data
.par_iter()
.zip(coefficients.par_iter())
.map(|(v, &c)| v.iter().map(|&x| x * c).sum::<i32>())
.collect();
}Reference handling differs based on whether you have owned or borrowed data.
use rayon::prelude::*;
fn benchmark_comparison() {
// Scenario 1: Large strings, transformation
let large_strings: Vec<String> = (0..10_000)
.map(|i| "x".repeat(i % 1000 + 1))
.collect();
// par_iter + clone: O(n) allocations for cloning
let result1: Vec<String> = large_strings
.par_iter()
.map(|s| s.to_uppercase())
.collect();
// into_par_iter: O(n) allocations for result (no clone)
let result2: Vec<String> = large_strings
.into_par_iter()
.map(|s| s.to_uppercase())
.collect();
// into_par_iter saves the clone allocation
// Scenario 2: Small Copy types
let small_ints: Vec<i32> = (0..100_000).collect();
// Difference is negligible for Copy types
let sum1: i32 = small_ints.par_iter().sum();
let sum2: i32 = small_ints.clone().into_par_iter().sum();
// Copy is cheap, no allocation difference
}For non-Copy types, into_par_iter eliminates clone overhead.
use rayon::prelude::*;
fn zipping() {
let a = vec![1, 2, 3];
let b = vec![4, 5, 6];
// Both borrowed
let sums: Vec<i32> = a.par_iter()
.zip(b.par_iter())
.map(|(&x, &y)| x + y)
.collect();
// One owned, one borrowed
let products: Vec<i32> = a.into_par_iter()
.zip(b.par_iter())
.map(|(x, &y)| x * y) // x is owned, y is borrowed
.collect();
// Both owned
let combined: Vec<(i32, i32)> = a.into_par_iter()
.zip(b.into_par_iter())
.collect();
}
fn zipping_strings() {
let names = vec!["Alice".to_string(), "Bob".to_string()];
let ids = vec![1, 2];
// Borrowing both - need to clone for owned output
let pairs: Vec<(String, i32)> = names
.par_iter()
.zip(ids.par_iter())
.map(|(name, &id)| (name.clone(), id))
.collect();
// Consuming one - no clone needed
let pairs: Vec<(String, i32)> = names
.into_par_iter()
.zip(ids.par_iter())
.map(|name, &id| (name, id)) // name is owned, no clone
.collect();
}Zipping combines iterators with matching ownership semantics.
use rayon::prelude::*;
use std::collections::{HashMap, HashSet, BTreeMap};
fn collect_types() {
let pairs = vec![(1, "a"), (2, "b"), (3, "c")];
// Collect into HashMap
let map: HashMap<i32, &str> = pairs
.par_iter()
.map(|&(k, v)| (k, v))
.collect();
// Collect into HashSet (keys only)
let set: HashSet<i32> = pairs
.par_iter()
.map(|&(k, _)| k)
.collect();
// With into_par_iter - owned values
let owned_pairs: Vec<(i32, String)> = pairs
.into_par_iter()
.map(|(k, v)| (k, v.to_uppercase()))
.collect();
// Into BTreeMap for sorted keys
let btree: BTreeMap<i32, String> = owned_pairs
.into_par_iter()
.collect();
}into_par_iter enables transformations that produce owned values during collection.
use rayon::prelude::*;
fn filter_map_collect() {
let data = vec![
"apple".to_string(),
"banana".to_string(),
"cherry".to_string(),
"date".to_string(),
];
// With par_iter: must clone
let long_upper: Vec<String> = data
.par_iter()
.filter(|s| s.len() > 5)
.map(|s| s.to_uppercase())
.collect();
// data still available
// With into_par_iter: no clone
let long_upper: Vec<String> = data
.into_par_iter()
.filter(|s| s.len() > 5)
.map(|s| s.to_uppercase())
.collect();
// data consumed
}
fn transform_and_preserve() {
let data = vec![1, 2, 3, 4, 5];
// Need original data? Use par_iter
let doubled: Vec<i32> = data
.par_iter()
.map(|&x| x * 2)
.collect();
// Can still use data
let sum: i32 = data.par_iter().sum();
// Don't need original? Use into_par_iter
let tripled: Vec<i32> = data
.into_par_iter()
.map(|x| x * 3)
.collect();
}Choose based on whether you need the original collection afterward.
use rayon::prelude::*;
fn comparison_table() {
// Summary of differences:
//
// par_iter():
// - Borrows collection
// - Yields references (&T)
// - Collection usable after iteration
// - Multiple passes possible
// - May need .cloned() for owned items
// - Works with other borrowed data
// par_iter_mut():
// - Borrows collection mutably
// - Yields mutable references (&mut T)
// - Collection usable after iteration
// - In-place modification
// - No allocation for new collection
// into_par_iter():
// - Consumes collection
// - Yields owned items (T)
// - Collection NOT usable after
// - Single pass only
// - No cloning needed
// - Better for transformations
// into_par_iter_mut():
// - Doesn't exist (use into_par_iter)
}Each variant serves different ownership needs.
use rayon::prelude::*;
struct Record {
id: u32,
name: String,
values: Vec<f64>,
}
fn process_pipeline() {
let records: Vec<Record> = (0..1000)
.map(|i| Record {
id: i,
name: format!("record_{}", i),
values: (0..10).map(|j| j as f64 * 0.1).collect(),
})
.collect();
// Scenario: Extract and transform, don't need original
let summaries: Vec<(u32, String, f64)> = records
.into_par_iter()
.map(|rec| {
let sum: f64 = rec.values.iter().sum();
(rec.id, rec.name.to_uppercase(), sum)
})
.collect();
// summaries is available, records is consumed
}
fn multi_pass_analysis() {
let records: Vec<Record> = (0..1000)
.map(|i| Record {
id: i,
name: format!("record_{}", i),
values: (0..10).map(|j| j as f64 * 0.1).collect(),
})
.collect();
// Need multiple passes? Use par_iter
let count = records.par_iter().count();
let avg_value: f64 = records
.par_iter()
.flat_map(|r| r.values.iter())
.sum::<f64>() / (count * 10) as f64;
let max_id = records.par_iter().map(|r| r.id).max().unwrap();
// Records still available for further use
println!("Count: {}, Avg: {}, Max ID: {}", count, avg_value, max_id);
}Pipeline design depends on whether the original data is needed later.
use rayon::prelude::*;
struct Pixel {
r: u8,
g: u8,
b: u8,
}
impl Pixel {
fn to_grayscale(&self) -> u8 {
(0.299 * self.r as f64 + 0.587 * self.g as f64 + 0.114 * self.b as f64) as u8
}
}
fn image_processing() {
let mut pixels: Vec<Pixel> = (0..100_000)
.map(|i| Pixel {
r: (i % 256) as u8,
g: ((i + 64) % 256) as u8,
b: ((i + 128) % 256) as u8,
})
.collect();
// In-place modification: par_iter_mut
pixels.par_iter_mut().for_each(|p| {
let gray = p.to_grayscale();
p.r = gray;
p.g = gray;
p.b = gray;
});
// Need original? Transform to new collection with par_iter
let original = vec![Pixel { r: 255, g: 0, b: 0 }; 100];
let grayscale: Vec<u8> = original
.par_iter()
.map(|p| p.to_grayscale())
.collect();
// original still available
// Don't need original? Use into_par_iter
let grayscale: Vec<u8> = original
.into_par_iter()
.map(|p| p.to_grayscale())
.collect();
}Image processing often uses par_iter_mut for in-place modifications.
Method comparison:
| Method | Ownership | Item Type | Collection After |
|--------|-----------|-----------|------------------|
| par_iter() | Borrowed | &T | Available |
| par_iter_mut() | Mutably borrowed | &mut T | Available |
| into_par_iter() | Consumed | T | Unavailable |
Use case matrix:
| Need | Recommended Method |
|------|-------------------|
| Multiple passes | par_iter() |
| Read-only access | par_iter() |
| In-place modification | par_iter_mut() |
| Transform and discard | into_par_iter() |
| Avoid cloning | into_par_iter() |
| Work with references | par_iter() |
| Extract owned fields | into_par_iter() |
Performance considerations:
| Factor | par_iter() | into_par_iter() |
|--------|--------------|-------------------|
| Clone overhead | May need .cloned() | No clone needed |
| Memory allocation | Additional for clones | Can reuse allocation |
| Copy types | Minimal difference | Minimal difference |
| Non-Copy types | Clone cost | Zero copy |
Key insight: The choice between into_par_iter() and par_iter() mirrors Rust's ownership model applied to parallel execution. Use par_iter() when you need to preserve the collection for subsequent operations or when performing read-only computationsāit borrows the collection and yields references, allowing multiple parallel passes. Use into_par_iter() when transforming data into a new form where the original collection is no longer neededāit consumes the collection and yields owned items, eliminating the need to clone non-Copy types. Use par_iter_mut() when you need in-place modifications without allocating a new collection. For non-Copy types like String or Vec, into_par_iter() can significantly reduce allocations by avoiding clones, while for Copy types like i32, the performance difference is negligible. The decision should be guided by ownership semantics: if you'd use iter() in sequential code, use par_iter() in parallel code; if you'd use into_iter(), use into_par_iter().