Loading page…
Rust walkthroughs
Loading page…
rayon::iter::IndexedParallelIterator::enumerate provide indices without requiring sequential iteration?IndexedParallelIterator::enumerate leverages the fact that indexed iterators know their exact length and can split at arbitrary positions, allowing each parallel chunk to calculate its starting index from its position in the split rather than counting sequentially. When a parallel iterator splits for execution across threads, the split position determines the starting index for that chunk, so each thread can enumerate its items independently using starting_index + local_offset. This is fundamentally different from sequential enumerate which must count through all prior items to know the current index.
fn sequential_enumerate() {
let data = vec
!["a", "b", "c", "d", "e"];
// Sequential enumerate must count through items
for (index, item) in data.iter().enumerate() {
println!("{}: {}", index, item);
}
// The iterator maintains internal state:
// - Start at index 0
// - Increment after each next() call
// - Must process items in order to maintain correct index
}Sequential enumerate increments an internal counter on each next() call, requiring items to be processed in order.
use rayon::prelude::*;
fn parallel_enumerate() {
let data = vec
!["a", "b", "c", "d", "e"];
// Parallel enumerate calculates indices from position
data.par_iter().enumerate().for_each(|(index, item)| {
println!("{}: {}", index, item);
});
// No sequential counting needed
// Each parallel chunk knows its starting position
}Parallel enumerate calculates indices from the known split position, enabling true parallelism.
use rayon::prelude::*;
fn indexed_trait() {
// IndexedParallelIterator requires:
// - len() -> usize: known length
// - split_at(self, index: usize) -> (Self, Self): split at position
// This is more powerful than ParallelIterator:
// - ParallelIterator: can only produce items
// - IndexedParallelIterator: knows length and can split at any point
let data = vec
![1, 2, 3, 4, 5, 6, 7, 8];
// par_iter() returns IndexedParallelIterator for slices
// Because slices know their length and can split at any index
let iter = data.par_iter();
// enumerate() works because it's an IndexedParallelIterator
iter.enumerate().for_each(|(i, &x)| {
println!("Index {} has value {}", i, x);
});
}IndexedParallelIterator provides the length and split-at capability needed for parallel enumeration.
use rayon::prelude::*;
fn split_at_mechanism() {
// Conceptual understanding of split_at:
let data = vec
![10, 20, 30, 40, 50, 60];
// When rayon splits work:
// 1. Thread pool decides how to divide work
// 2. split_at is called with an index
// 3. Two iterators are created
// Example: split_at(3) on a 6-element slice
// let (left, right) = iter.split_at(3);
// left: [10, 20, 30] with indices 0, 1, 2
// right: [40, 50, 60] with indices 3, 4, 5
// For enumerate:
// - left chunk starts at index 0
// - right chunk starts at index 3 (split position)
// Each chunk enumerates locally:
// left: 0+10, 1+20, 2+30
// right: 3+40, 4+50, 5+60
// The split position IS the starting index for the right chunk
}The split position directly determines the starting index for each chunk.
use rayon::prelude::*;
// Conceptual structure of enumerate for IndexedParallelIterator:
//
// struct Enumerate<I> {
// base: I,
// }
//
// impl<I: IndexedParallelIterator> ParallelIterator for Enumerate<I> {
// type Item = (usize, I::Item);
//
// fn drive_unindexed(self, consumer: C) -> C::Result {
// // Delegate to drive with index information
// }
// }
//
// When split_at is called on Enumerate:
// 1. Split the base iterator at index
// 2. The left part gets indices 0..split_index
// 3. The right part gets indices split_index..len
fn implementation_example() {
let data: Vec<i32> = (0..1000).collect();
// enumerate internally:
// 1. Knows total length = 1000
// 2. When work is divided (e.g., 4 threads):
// - Thread 1: split at 250, indices 0..250
// - Thread 2: split at 250, indices 250..500
// - Thread 3: split at 500, indices 500..750
// - Thread 4: split at 750, indices 750..1000
data.par_iter().enumerate().for_each(|(index, &value)| {
// Each thread calculates: base_index + local_offset
// No communication between threads needed
});
}Each parallel chunk independently calculates indices from its known position.
use rayon::prelude::*;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
fn comparison() {
let data: Vec<i32> = (0..10).collect();
// Sequential: must process in order
let mut sequential_order = Vec::new();
for (i, _) in data.iter().enumerate() {
sequential_order.push(i);
}
assert_eq!(sequential_order, vec
![0, 1, 2, 3, 4, 5, 6, 7, 8, 9]);
// Parallel: can process out of order, but indices are still correct
let counter = Arc::new(AtomicUsize::new(0));
data.par_iter().enumerate().for_each(|(i, _)| {
// i is always correct, even though threads run in parallel
// No race condition on the index
});
// Sequential enumerate needs:
// - Single counter
// - Increment after each item
// - Must process items in sequence
// Parallel enumerate needs:
// - Known length
// - Split position
// - Each chunk calculates its own indices
}Parallel enumerate produces correct indices without sequential processing.
use rayon::prelude::*;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
fn parallel_chunks() {
let data: Vec<i32> = (0..100).collect();
let max_index = Arc::new(AtomicU64::new(0));
let min_index = Arc::new(AtomicU64::new(100));
data.par_iter().enumerate().for_each(|(index, _)| {
// Each thread gets the correct index without coordination
// This works because indices are calculated from position
max_index.fetch_max(index as u64, Ordering::Relaxed);
min_index.fetch_min(index as u64, Ordering::Relaxed);
});
assert_eq!(max_index.load(Ordering::Relaxed), 99);
assert_eq!(min_index.load(Ordering::Relaxed), 0);
}All indices from 0 to len-1 appear exactly once, regardless of parallel execution order.
use rayon::prelude::*;
fn why_indexed_required() {
// ParallelIterator (not indexed) can:
// - Produce items in parallel
// - Process items with for_each, map, etc.
// - But does NOT know length or support split_at
// Example: A parallel iterator from a channel
// Items arrive unpredictably
// No known length
// Cannot split at arbitrary positions
// Such iterators CANNOT use enumerate because:
// - No way to know how many items before current one
// - Cannot calculate position without counting sequentially
// IndexedParallelIterator is required for enumerate:
// - Knows exact length
// - Can split at any position
// - Position determines index directly
}Non-indexed parallel iterators cannot use enumerate because they lack length and split-at capabilities.
use rayon::prelude::*;
fn recursive_splits() {
let data: Vec<i32> = (0..16).collect();
// Rayon recursively splits work:
// Level 0: [0..16], indices 0..16
// Level 1: [0..8] [8..16], indices 0..8 and 8..16
// Level 2: [0..4] [4..8] [8..12] [12..16]
// ...and so on until chunks are small enough
data.par_iter().enumerate().for_each(|(index, &value)| {
// Each leaf chunk knows its starting index
// No matter how deep the split tree
// The index is: split_position + offset_within_chunk
});
// The split position propagates down:
// - Top level: split at 8, right half starts at 8
// - Next level: right half splits at 4 (relative),
// which is 8+4=12 (absolute)
}Recursive splits propagate position information down to each chunk.
use rayon::prelude::*;
fn different_iterators() {
// Slices
let slice = &[1, 2, 3, 4, 5];
slice.par_iter().enumerate().for_each(|(i, _)| {
// i goes from 0 to 4
});
// Vec
let vec = vec
![1, 2, 3, 4, 5];
vec.par_iter().enumerate().for_each(|(i, _)| {
// i goes from 0 to 4
});
// Range
(0..100).into_par_iter().enumerate().for_each(|(i, _)| {
// i goes from 0 to 99
});
// All implement IndexedParallelIterator:
// - Known length
// - Can split at any position
// - enumerate works correctly
}Any IndexedParallelIterator can use enumerate correctly in parallel.
use rayon::prelude::*;
fn after_transformations() {
let data: Vec<i32> = (0..10).collect();
// map preserves indexed property
data.par_iter()
.map(|x| x * 2)
.enumerate()
.for_each(|(index, value)| {
// index corresponds to original position
// value is the transformed value
println!("Index {}: Value {}", index, value);
});
// filter DOES NOT preserve indexed property
// This would NOT compile:
// data.par_iter()
// .filter(|x| *x % 2 == 0)
// .enumerate() // Error: not IndexedParallelIterator
// .for_each(|(i, v)| {});
// After filter, length is unknown
// Cannot split at positions
// enumerate is not available
}Transformations like map preserve the indexed property; filter does not.
use rayon::prelude::*;
fn preserving_index() {
// Operations that preserve IndexedParallelIterator:
// - map, map_with, map_init
// - cloned, copied
// - inspect
// - update
// - enumerate (returns IndexedParallelIterator)
// - skip (only if skip amount is known at compile time for some cases)
// Operations that lose IndexedParallelIterator:
// - filter, filter_map
// - flat_map, flatten
// - while the output might have indexed, the operation may lose it
// - any operation that changes the number of items unpredictably
let data: Vec<i32> = (0..10).collect();
// Chain of indexed-preserving operations:
data.par_iter()
.cloned() // IndexedParallelIterator
.map(|x| x * 2) // IndexedParallelIterator
.enumerate() // IndexedParallelIterator (yields (usize, i32))
.map(|(i, x)| (i, x + 1)) // IndexedParallelIterator
.for_each(|(index, value)| {
println!("{}: {}", index, value);
});
}Transformations that preserve length and position retain enumerate capability.
use rayon::prelude::*;
use std::time::Instant;
fn performance() {
let data: Vec<i32> = (0..1_000_000).collect();
// Sequential enumerate
let start = Instant::now();
let sum: i64 = data.iter()
.enumerate()
.map(|(i, _)| i as i64)
.sum();
let sequential_time = start.elapsed();
// Parallel enumerate
let start = Instant::now();
let sum: i64 = data.par_iter()
.enumerate()
.map(|(i, _)| i as i64)
.sum();
let parallel_time = start.elapsed();
// Parallel version:
// - No atomic counter for indices
// - No synchronization between threads
// - Each thread calculates indices from position
// - Scales with number of cores
// The index calculation is O(1) per chunk:
// index = base_index + local_offset
// where base_index is the split position
}Parallel enumerate has no synchronization overhead for index tracking.
use rayon::prelude::*;
fn work_stealing() {
// Rayon uses work stealing for load balancing
// Even with work stealing, indices remain correct
let data: Vec<i32> = (0..100).collect();
// Initial split: 4 threads, ~25 items each
// Thread 1: 0..25, Thread 2: 25..50, etc.
// If Thread 1 finishes early and steals from Thread 2:
// - Stolen chunk is a sub-range of Thread 2's work
// - The split position for stolen work is known
// - Indices are calculated from that position
// Example:
// - Thread 2 has indices 25..50
// - Steals half: Thread 2 has 25..37, Thread 1 has 37..50
// - Both know their indices from the split position
data.par_iter().enumerate().for_each(|(i, _)| {
// Even with work stealing, i is always correct
});
}Work stealing preserves correct indices because stolen chunks know their position.
use rayon::prelude::*;
fn parallel_with_indices() {
let data: Vec<String> = (0..100)
.map(|i| format!("item_{}", i))
.collect();
// Process items in parallel, need indices for output
let results: Vec<(usize, String)> = data.par_iter()
.enumerate()
.map(|(index, item)| {
// Index is available for processing
let processed = format!("processed_{}_{}", index, item);
(index, processed)
})
.collect();
// Results are in arbitrary order (parallel execution)
// But indices are always correct
// Sort by index to restore order
let mut sorted_results: Vec<_> = results.into_iter().collect();
sorted_results.sort_by_key(|(index, _)| *index);
for (index, value) in sorted_results {
assert!(value.starts_with(&format!("processed_{}_", index)));
}
}Use indices to track or restore order when processing in parallel.
use rayon::prelude::*;
fn enumerate_collect() {
let data: Vec<i32> = (0..5).collect();
// enumerate returns (usize, &i32)
let indexed: Vec<(usize, &i32)> = data.par_iter()
.enumerate()
.collect();
// Collected order is undefined (parallel execution)
// But all index-value pairs are present
// Verify all indices are present
let indices: std::collections::HashSet<usize> =
indexed.iter().map(|(i, _)| *i).collect();
assert_eq!(indices.len(), 5);
assert!(indices.contains(&0));
assert!(indices.contains(&4));
}Collected results may be in arbitrary order, but all indices appear exactly once.
use rayon::prelude::*;
fn nested_enumerate() {
let matrix: Vec<Vec<i32>> = vec
![
vec
![1, 2, 3],
vec
![4, 5, 6],
vec
![7, 8, 9],
];
// Enumerate outer dimension
matrix.par_iter()
.enumerate()
.for_each(|(row_idx, row)| {
// row_idx is the row index
// Enumerate inner dimension
row.par_iter()
.enumerate()
.for_each(|(col_idx, &value)| {
// col_idx is the column index
println!("matrix[{}][{}] = {}", row_idx, col_idx, value);
});
});
// Each level of enumerate works independently
// Row indices: 0, 1, 2 (calculated from outer split positions)
// Column indices: 0, 1, 2 (calculated from inner split positions)
}Nested enumerate works correctly for multi-dimensional data.
use rayon::prelude::*;
fn sequential_vs_parallel_indices() {
let data: Vec<i32> = (0..100).collect();
// Sequential: indices are generated in order
let sequential_indices: Vec<usize> = data.iter()
.enumerate()
.map(|(i, _)| i)
.collect();
assert_eq!(sequential_indices, (0..100).collect::<Vec<_>>());
// Parallel: indices are correct but execution order varies
let parallel_indices: Vec<usize> = data.par_iter()
.enumerate()
.map(|(i, _)| i)
.collect();
// Same indices, potentially different order
let mut sorted_parallel: Vec<_> = parallel_indices.into_iter().collect();
sorted_parallel.sort();
assert_eq!(sorted_parallel, (0..100).collect::<Vec<_>>());
}Both produce correct indices; sequential has defined order, parallel may vary.
Key insight: IndexedParallelIterator::enumerate doesn't count through items sequentially—instead, it uses the split position when work is divided to calculate the starting index for each chunk.
How it works:
// Sequential enumerate:
// - Maintains a counter starting at 0
// - Increments counter after each next()
// - Must process items in sequence
// Parallel enumerate (IndexedParallelIterator):
// - Knows total length: len()
// - When split_at(index) is called:
// - Left chunk: indices 0 to index-1
// - Right chunk: indices index to len-1
// - Each chunk calculates: base_index + local_offset
// - No coordination between threads neededWhy this is efficient:
// No atomic operations for counting
// No synchronization between threads
// Each thread has complete information for its chunk
// Split position directly maps to index
// O(1) index calculation per splitRequirements:
// IndexedParallelIterator requires:
// 1. len() -> usize: known total length
// 2. split_at(index) -> (Self, Self): split at position
//
// These allow enumerate to work without sequential counting
// The split position IS the starting index for the right chunkThe fundamental insight is that for an indexed parallel iterator, the position in the sequence is known without counting—splitting at position 100 means the second chunk starts at index 100. This is why IndexedParallelIterator::enumerate can provide correct indices in parallel: each split knows its position, and that position is the starting index for enumeration.