Loading page…
Rust walkthroughs
Loading page…
tempfile::SpooledTempFile and NamedTempFile for memory-constrained temporary storage?SpooledTempFile stores data in memory up to a configurable threshold before spilling to disk, while NamedTempFile immediately creates a file on the filesystem with a persistent path. The trade-off centers on the memory-I/O balance: SpooledTempFile optimizes for small temporary data by avoiding filesystem operations entirely until necessary, whereas NamedTempFile provides immediate filesystem presence for scenarios requiring a stable path or larger data. For memory-constrained environments, SpooledTempFile with a low threshold gracefully handles both small in-memory data and large spillover without pre-allocating disk space, while NamedTempFile requires filesystem availability from the start but offers predictable resource usage.
use tempfile::NamedTempFile;
use std::io::Write;
fn main() -> std::io::Result<()> {
// Creates file immediately on disk
let mut temp_file = NamedTempFile::new()?;
// File has a stable path from creation
let path = temp_file.path();
println!("Created at: {:?}", path);
// Write data - goes directly to disk
temp_file.write_all(b"Hello, temporary world!")?;
// Persist the file (otherwise deleted on drop)
let persisted_path = temp_file.into_temp_path();
println!("Persisted at: {:?}", persisted_path);
Ok(())
}NamedTempFile creates a real file from the moment of instantiation.
use tempfile::SpooledTempFile;
use std::io::Write;
fn main() -> std::io::Result<()> {
// Creates in-memory buffer, no filesystem access yet
let mut temp = SpooledTempFile::new(1024); // 1KB threshold
// Small data stays in memory
temp.write_all(b"Small data")?;
println!("In memory, no disk I/O yet");
// Data stays in memory until threshold exceeded
temp.write_all(&vec![0u8; 2000])?; // Exceeds 1KB
// Now spilled to disk automatically
println!("Spilled to disk after exceeding threshold");
// Can check if spilled
if temp.is_rolled() {
println!("Data is now on disk");
}
Ok(())
}SpooledTempFile starts in memory and transitions to disk when size exceeds the threshold.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
fn main() -> std::io::Result<()> {
// NamedTempFile: Immediate disk allocation
let named = NamedTempFile::new()?;
// - Creates file in temp directory immediately
// - Uses filesystem space from the start
// - No memory buffer for content
drop(named); // File deleted
// SpooledTempFile: Configurable memory budget
let mut spooled = SpooledTempFile::new(5_000_000); // 5MB threshold
// Memory usage pattern:
// - 0 to 5MB: Pure in-memory storage
// - Over 5MB: Spills to disk, frees memory
// Write 3MB - stays in memory
spooled.write_all(&vec![0u8; 3_000_000])?;
assert!(!spooled.is_rolled());
// Write another 3MB - exceeds 5MB, spills to disk
spooled.write_all(&vec![0u8; 3_000_000])?;
assert!(spooled.is_rolled());
// Memory is freed after spilling
// Only small buffer overhead remains
Ok(())
}SpooledTempFile provides a bounded memory budget; NamedTempFile has no memory component.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
use std::path::Path;
fn main() -> std::io::Result<()> {
// NamedTempFile: Always has a path
let named = NamedTempFile::new()?;
let named_path = named.path();
// Can share with other processes
println!("Path available: {:?}", named_path);
// Other process can open this file
// Useful for inter-process communication
// SpooledTempFile: No path until spilled
let mut spooled = SpooledTempFile::new(1000);
// No path available while in memory
// spooled.path() doesn't exist
// After spilling:
spooled.write_all(&vec![0u8; 2000])?; // Exceeds threshold
if spooled.is_rolled() {
// Now has a path on disk
// But by then, file already exists
println!("Spilled - now has disk presence");
}
// Key difference:
// NamedTempFile: Path available immediately
// SpooledTempFile: Path only available after spilling
Ok(())
}NamedTempFile provides immediate path access; SpooledTempFile only after spilling.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::{Write, Read, Seek, SeekFrom};
use std::time::Instant;
fn main() -> std::io::Result<()> {
let data = vec![0u8; 100_000]; // 100KB
// NamedTempFile: Always disk operations
let start = Instant::now();
let mut named = NamedTempFile::new()?;
named.write_all(&data)?;
let named_write = start.elapsed();
named.seek(SeekFrom::Start(0))?;
let mut buf = vec![0u8; 100_000];
named.read_exact(&mut buf)?;
let named_read = start.elapsed();
// SpooledTempFile: Memory operations if under threshold
let start = Instant::now();
let mut spooled = SpooledTempFile::new(200_000); // 200KB threshold
spooled.write_all(&data)?;
let spooled_write = start.elapsed();
// Much faster - no disk I/O
println!("NamedTempFile write: {:?}", named_write);
println!("SpooledTempFile write: {:?}", spooled_write);
// Reading from memory is also faster
let start = Instant::now();
spooled.seek(SeekFrom::Start(0))?;
let mut buf2 = vec![0u8; 100_000];
spooled.read_exact(&mut buf2)?;
let spooled_read = start.elapsed();
println!("NamedTempFile read: {:?}", named_read);
println!("SpooledTempFile read: {:?}", spooled_read);
Ok(())
}SpooledTempFile avoids disk latency for small data; NamedTempFile always pays disk overhead.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
fn main() -> std::io::Result<()> {
// Both types delete on drop by default
// NamedTempFile: Can persist file
let mut named = NamedTempFile::new()?;
named.write_all(b"persistent data")?;
let persisted = named.persist("/path/to/final/location")?;
// File now exists at final location, temp deleted
// SpooledTempFile: Can also persist after spill
let mut spooled = SpooledTempFile::new(100)?;
spooled.write_all(&vec![0u8; 200])?; // Force spill
// After spilling, behaves like NamedTempFile
if spooled.is_rolled() {
let (file, path) = spooled.into_parts();
// Can work with underlying file
println!("File at: {:?}", path);
}
// For in-memory SpooledTempFile (not spilled):
let mut small = SpooledTempFile::new(1000)?;
small.write_all(b"tiny")?;
// into_parts returns different type
let content = small.into_inner()?;
// content is Vec<u8> (the in-memory buffer)
// No file was ever created on disk
Ok(())
}Cleanup behavior differs based on whether SpooledTempFile has spilled.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
fn main() -> std::io::Result<()> {
// NamedTempFile: Requires filesystem
// - Fails if temp directory unavailable
// - Fails if filesystem full
// - Fails if permissions insufficient
// This could fail in restricted environments:
// let named = NamedTempFile::new()?; // Needs writable temp dir
// SpooledTempFile: No filesystem needed (until spill)
let mut spooled = SpooledTempFile::new(1000)?;
spooled.write_all(b"small data")?;
// No filesystem access required!
// Works even if:
// - Temp directory doesn't exist
// - Filesystem is read-only
// - No disk space available
// Only fails when spilling and filesystem unavailable:
// (Would fail if threshold exceeded and no temp dir)
// Useful for environments with restricted filesystem access
println!("SpooledTempFile works without filesystem");
Ok(())
}SpooledTempFile works without filesystem access until the threshold is exceeded.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
// Guideline for choosing:
//
// Data Size | Recommended Type | Reason
// -------------|----------------------|--------------------------------
// < 1MB | SpooledTempFile | Memory operations much faster
// 1-10MB | SpooledTempFile | Good balance, spill is rare
// 10-100MB | Depends on memory | Consider available RAM
// > 100MB | NamedTempFile | Large data, disk inevitable
// Unknown | SpooledTempFile | Graceful degradation
fn process_small_data() -> std::io::Result<()> {
// Small data: SpooledTempFile avoids disk entirely
let mut temp = SpooledTempFile::new(1024 * 1024)?; // 1MB threshold
temp.write_all(b"Small payload")?;
// Fast in-memory operations
Ok(())
}
fn process_large_data() -> std::io::Result<()> {
// Large data: NamedTempFile avoids memory pressure
let mut temp = NamedTempFile::new()?;
temp.write_all(&vec![0u8; 100_000_000])?; // 100MB
// Goes directly to disk, no memory spike
Ok(())
}
fn process_unknown_size() -> std::io::Result<()> {
// Unknown size: SpooledTempFile adapts
let mut temp = SpooledTempFile::new(10_000_000)?; // 10MB threshold
// If small: stays in memory (fast)
// If large: spills to disk (graceful)
// Streaming data:
let mut received = 0;
loop {
let chunk = vec![0u8; 1000]; // Simulated incoming chunk
temp.write_all(&chunk)?;
received += chunk.len();
// Can check if spilled to adapt behavior
if temp.is_rolled() {
println!("Spilled at {} bytes", received);
break;
}
if received >= 50_000 {
break;
}
}
Ok(())
}
fn main() {
process_small_data().unwrap();
process_large_data().unwrap();
process_unknown_size().unwrap();
}Choose based on expected data size and memory availability.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::{Write, Seek, SeekFrom, Read};
fn main() -> std::io::Result<()> {
// NamedTempFile: Always supports seeking
let mut named = NamedTempFile::new()?;
named.write_all(b"Hello World")?;
named.seek(SeekFrom::Start(0))?;
let mut buf = [0u8; 5];
named.read_exact(&mut buf)?;
println!("Read: {:?}", std::str::from_utf8(&buf)); // "Hello"
// SpooledTempFile: Also supports seeking
let mut spooled = SpooledTempFile::new(100)?;
spooled.write_all(b"Hello World")?;
// Seeking works in memory
spooled.seek(SeekFrom::Start(6))?;
let mut buf2 = [0u8; 5];
spooled.read_exact(&mut buf2)?;
println!("Read: {:?}", std::str::from_utf8(&buf2)); // "World"
// Also works after spilling
spooled.write_all(&vec![0u8; 200])?; // Force spill
assert!(spooled.is_rolled());
spooled.seek(SeekFrom::Start(0))?;
// Now seeking on underlying file
Ok(())
}Both types support seeking; SpooledTempFile uses memory seek initially.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
use std::fs::File;
fn main() -> std::io::Result<()> {
// NamedTempFile: Can be shared via path
let named = NamedTempFile::new()?;
let path = named.path().to_pathBuf();
// Another process/thread can open the file:
// let another = File::open(&path)?;
// (NamedTempFile manages exclusive access)
// SpooledTempFile: Cannot be shared while in memory
let spooled = SpooledTempFile::new(1000)?;
// No path available - cannot share
// Exclusive access to memory buffer
// After spilling, has path like NamedTempFile
// But by then it's already on disk
// For inter-process communication:
// Use NamedTempFile for immediate visibility
// Use SpooledTempFile for private temporary storage
Ok(())
}NamedTempFile supports sharing via path; SpooledTempFile is private until spilled.
use tempfile::{NamedTempFile, SpooledTempFile};
use std::io::Write;
fn main() -> std::io::Result<()> {
// Scenario: Memory-constrained environment
// Option 1: NamedTempFile with controlled size
// - Predictable: only uses filesystem
// - No memory spikes
// - But requires filesystem from start
// Option 2: SpooledTempFile with low threshold
let mut temp = SpooledTempFile::new(1024)?; // 1KB threshold
// Benefits:
// - Small data stays in memory (fast)
// - Large data spills to disk (bounded memory)
// - Adapts to actual data size
temp.write_all(b"short text")?; // In memory (9 bytes)
assert!(!temp.is_rolled());
temp.write_all(&vec![0u8; 5000])?; // Spills (> 1KB)
assert!(temp.is_rolled());
// Memory usage at peak:
// - Before spill: ~1KB buffer
// - After spill: buffer freed, only disk used
// Compare to NamedTempFile:
// - Always creates file
// - No memory buffer
// - But immediate filesystem dependency
// For truly constrained memory:
// Use SpooledTempFile with very low threshold (or 0)
let mut ultra_constrained = SpooledTempFile::new(0)?; // Immediate spill
// This essentially becomes NamedTempFile behavior
// But you still control when spill happens
Ok(())
}In memory-constrained environments, SpooledTempFile with a low threshold provides bounded memory usage.
use tempfile::SpooledTempFile;
use std::io::Write;
fn main() -> std::io::Result<()> {
// SpooledTempFile works well with streaming operations
// Compression: Data shrinks during processing
// Keep compressed data in memory, spill uncompressed
let mut compressed_temp = SpooledTempFile::new(1024 * 1024)?;
// Encryption: Stream encryption
// Small encrypted blobs in memory, large spills
// The key insight: SpooledTempFile adapts to actual size
// Not pre-allocated like NamedTempFile
// Example: Variable-size data
let mut temp = SpooledTempFile::new(10_000_000)?;
// Process incoming stream
for chunk in [
b"chunk1".to_vec(),
b"chunk2".to_vec(),
b"chunk3".to_vec(),
] {
temp.write_all(&chunk)?;
// If small: stays in memory
// If large: spills automatically
// No need to predict final size
}
Ok(())
}SpooledTempFile adapts to variable-size data without pre-allocation.
// | Aspect | NamedTempFile | SpooledTempFile |
// |----------------------|----------------------------|---------------------------|
// | Initial storage | Disk (file created) | Memory (no file) |
// | Path available | Immediately | Only after spill |
// | Memory usage | Minimal (no buffer) | Up to threshold |
// | Disk usage | From creation | Only after spill |
// | Filesystem required | Yes, from start | No, until spill |
// | Performance (small) | Slower (disk I/O) | Fast (memory) |
// | Performance (large) | Similar | Similar (disk after spill)|
// | Cleanup | On drop or persist | On drop (memory or file) |
// | Use case | Need path, large data | Small data, unknown size |
// | Sharing | Path can be shared | Private until spill |use tempfile::{NamedTempFile, SpooledTempFile};
// Decision tree:
//
// 1. Do you need a path immediately (for other processes, APIs)?
// -> NamedTempFile
//
// 2. Is data size reliably small (< available memory)?
// -> SpooledTempFile with appropriate threshold
//
// 3. Is data size reliably large (> threshold you'd set)?
// -> NamedTempFile (avoids memory overhead)
//
// 4. Is data size unknown or variable?
// -> SpooledTempFile (adapts to size)
//
// 5. Is filesystem access restricted or unreliable?
// -> SpooledTempFile (works in memory until spill)
//
// 6. Need predictable resource usage?
// -> NamedTempFile (always uses disk)
//
// 7. Want to minimize disk operations for small data?
// -> SpooledTempFile (memory-first)
fn main() {
// Example: API response caching
// Use SpooledTempFile - responses are usually small
// Example: Video processing intermediate files
// Use NamedTempFile - videos are large
// Example: Log file rotation
// Use SpooledTempFile - can buffer in memory before disk
// Example: Inter-process communication file
// Use NamedTempFile - need path immediately
}Choose based on whether you need immediate path access, expected data size, and filesystem constraints.
Key trade-offs:
| Factor | NamedTempFile | SpooledTempFile | |--------|---------------|-----------------| | Startup cost | Filesystem operation | None | | Memory footprint | Minimal | Bounded by threshold | | Path availability | Immediate | After spill only | | Small data speed | Disk latency | Memory speed | | Large data handling | Immediate disk | Spills gracefully | | Filesystem dependency | Required | Optional until spill |
When to use NamedTempFile:
When to use SpooledTempFile:
Key insight: SpooledTempFile provides graceful degradation from memory to disk, making it ideal when data size is unpredictable. It avoids filesystem dependency for small data while handling large data without memory pressure. NamedTempFile trades memory efficiency for predictability—immediate path availability and guaranteed disk usage. For memory-constrained environments, SpooledTempFile with a carefully chosen threshold provides the best balance: small data benefits from memory speed while large data automatically overflows to disk without pre-allocation. The threshold should be set based on available memory and expected data size distribution, allowing the common case (small data) to stay in memory while gracefully handling the uncommon case (large data).