Loading page…
Rust walkthroughs
Loading page…
tokio::fs and std::fs for file operations in async code?tokio::fs provides an async API for file operations that internally uses spawn_blocking to offload blocking I/O to a thread pool, preventing it from blocking the async runtime. std::fs performs synchronous blocking I/O on the current thread. The fundamental trade-off is that tokio::fs keeps the async runtime responsive by moving blocking work elsewhere, but incurs overhead from thread pool scheduling and context switches. std::fs is simpler and has lower overhead per operation but will block the async runtime, preventing other tasks from making progress. For high-concurrency async applications, tokio::fs is appropriate; for batch processing or when you control thread allocation explicitly, std::fs with manual spawn_blocking may be more efficient.
use std::fs;
use std::io::Read;
fn main() -> std::io::Result<()> {
// std::fs blocks the current thread
let mut file = fs::File::open("data.txt")?;
let mut contents = String::new();
file.read_to_string(&mut contents)?;
// This thread is blocked during the read
// In async code, this blocks the entire executor
println!("Read {} bytes", contents.len());
Ok(())
}std::fs operations are blocking at the OS level.
use tokio::fs;
use tokio::io::AsyncReadExt;
#[tokio::main]
async fn main() -> std::io::Result<()> {
// tokio::fs provides async-style API
let mut file = fs::File::open("data.txt").await?;
let mut contents = String::new();
file.read_to_string(&mut contents).await?;
// The .await allows other tasks to run
// Internally, this uses spawn_blocking
println!("Read {} bytes", contents.len());
Ok(())
}tokio::fs provides .await points for async compatibility.
use tokio::task::spawn_blocking;
use std::fs;
// tokio::fs::File::open is roughly equivalent to:
async fn tokio_file_open(path: &str) -> std::io::Result<tokio::fs::File> {
let path = path.to_owned();
spawn_blocking(move || {
// This runs on the blocking thread pool
fs::File::open(&path)
}).await
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
// tokio::fs operations use spawn_blocking internally
// They offload to a separate thread pool
let contents = tokio::fs::read_to_string("data.txt").await?;
println!("Read {} bytes", contents.len());
Ok(())
}tokio::fs wraps blocking operations in spawn_blocking.
use std::fs;
use tokio::time::{sleep, Duration};
#[tokio::main]
async fn main() {
// Spawn a task that should print periodically
let ticker = tokio::spawn(async {
let mut count = 0;
loop {
sleep(Duration::from_millis(100)).await;
count += 1;
println!("Tick {}", count);
}
});
// This std::fs call blocks the runtime
// The ticker task cannot run during this
let _ = fs::read_to_string("large_file.txt");
// Ticks are delayed while reading the file
// The runtime cannot poll other tasks
ticker.abort();
}std::fs in async code blocks all tasks on the same thread.
use tokio::fs;
use tokio::time::{sleep, Duration};
#[tokio::main]
async fn main() {
// Spawn a task that prints periodically
let ticker = tokio::spawn(async {
let mut count = 0;
loop {
sleep(Duration::from_millis(100)).await;
count += 1;
println!("Tick {}", count);
}
});
// tokio::fs yields during the operation
// The ticker task continues running
let _ = fs::read_to_string("large_file.txt").await;
// Ticks continue while reading
// The runtime remains responsive
ticker.abort();
}tokio::fs keeps the runtime responsive during file operations.
use tokio::fs;
use std::fs;
use std::time::Instant;
#[tokio::main]
async fn main() -> std::io::Result<()> {
let iterations = 100;
// std::fs: direct call, no async overhead
let start = Instant::now();
for _ in 0..iterations {
let _ = fs::read_to_string("small.txt")?;
}
let std_time = start.elapsed();
// tokio::fs: spawn_blocking overhead
let start = Instant::now();
for _ in 0..iterations {
let _ = fs::read_to_string("small.txt").await?;
}
let tokio_time = start.elapsed();
println!("std::fs: {:?}", std_time);
println!("tokio::fs: {:?}", tokio_time);
// For small files and many operations, std::fs is faster
// The spawn_blocking overhead adds up
Ok(())
}std::fs has lower per-operation overhead than tokio::fs.
use tokio::task::spawn_blocking;
use std::fs;
#[tokio::main]
async fn main() -> std::io::Result<()> {
// Instead of individual tokio::fs calls
// Batch operations in one spawn_blocking
let results = spawn_blocking(move || {
// All blocking operations together
let data1 = fs::read_to_string("file1.txt")?;
let data2 = fs::read_to_string("file2.txt")?;
let data3 = fs::read_to_string("file3.txt")?;
Ok::<_, std::io::Error>((data1, data2, data3))
}).await??;
// Only one spawn_blocking call for multiple operations
// Reduces thread pool scheduling overhead
println!("Read {} bytes total",
results.0.len() + results.1.len() + results.2.len());
Ok(())
}Batching std::fs operations in spawn_blocking can be more efficient.
use tokio::runtime::Builder;
#[tokio::main]
async fn main() {
// Default: blocking thread pool has limited threads
// Defaults to num_cpus * 2 for max_blocking_threads
}
fn custom_runtime() -> tokio::runtime::Runtime {
Builder::new_multi_thread()
.worker_threads(4) // Async worker threads
.max_blocking_threads(16) // Blocking thread pool size
.build()
.unwrap()
}
// If you have many concurrent tokio::fs operations,
// you may need to increase max_blocking_threads
// Otherwise, operations queue up waiting for threadsBlocking thread pool size affects tokio::fs concurrency.
use tokio::fs;
use tokio::net::TcpListener;
use tokio::io::AsyncWriteExt;
#[tokio::main]
async fn main() -> std::io::Result<()> {
let listener = TcpListener::bind("127.0.0.1:8080").await?;
loop {
let (mut socket, _) = listener.accept().await?;
// Handle each connection in a task
tokio::spawn(async move {
// Read file to send
let content = match fs::read_to_string("response.txt").await {
Ok(c) => c,
Err(_) => return,
};
// Send to client
let _ = socket.write_all(content.as_bytes()).await;
});
}
}In servers, tokio::fs prevents one slow file read from blocking all connections.
use std::fs;
use tokio::task::spawn_blocking;
#[tokio::main]
async fn main() -> std::io::Result<()> {
// For startup/shutdown operations
// Where blocking briefly is acceptable
let config = fs::read_to_string("config.json")?;
println!("Loaded config");
// For batch file processing
// Where you want explicit control
let results = spawn_blocking(|| {
let mut total = 0;
for entry in fs::read_dir("data/")? {
let path = entry?.path();
if path.extension().map(|e| e == "txt").unwrap_or(false) {
let content = fs::read_to_string(&path)?;
total += content.lines().count();
}
}
Ok::<_, std::io::Error>(total)
}).await??;
println!("Total lines: {}", results);
Ok(())
}Use std::fs for startup, shutdown, or batch processing with explicit spawn_blocking.
use tokio::fs;
use tokio::task::JoinSet;
#[tokio::main]
async fn main() -> std::io::Result<()> {
let paths = vec!["file1.txt", "file2.txt", "file3.txt", "file4.txt"];
let mut tasks = JoinSet::new();
// Spawn concurrent reads
for path in paths {
tasks.spawn(async move {
fs::read_to_string(path).await
});
}
// Collect results as they complete
let mut contents = Vec::new();
while let Some(result) = tasks.join_next().await {
contents.push(result??);
}
println!("Read {} files", contents.len());
Ok(())
}tokio::fs enables true concurrency for multiple file operations.
use tokio::fs;
use std::fs;
#[tokio::main]
async fn main() {
// std::fs: synchronous Result
match std::fs::read_to_string("nonexistent.txt") {
Ok(content) => println!("Content: {}", content),
Err(e) => println!("Error: {}", e),
}
// tokio::fs: async Result (await gives io::Result)
match fs::read_to_string("nonexistent.txt").await {
Ok(content) => println!("Content: {}", content),
Err(e) => println!("Error: {}", e),
}
// Error types are the same (std::io::Error)
// tokio::fs wraps them in async machinery
}Both use std::io::Error; tokio::fs adds async wrapping.
use tokio::fs::File;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use std::fs::File as StdFile;
use std::io::{Read, Write};
#[tokio::main]
async fn main() -> std::io::Result<()> {
// std::fs::File: blocking file handle
let mut std_file = StdFile::open("data.txt")?;
let mut buffer = vec![0u8; 1024];
std_file.read(&mut buffer)?;
// File handle is blocking
// tokio::fs::File: async file handle
let mut tokio_file = File::open("data.txt").await?;
tokio_file.read(&mut buffer).await?;
// File handle has async methods
// Cannot use std::fs::File in async code without spawn_blocking
// Cannot use tokio::fs::File in blocking code
Ok(())
}File handles are different types; use the matching API.
use tokio::fs;
#[tokio::main]
async fn main() -> std::io::Result<()> {
// On Linux/macOS: file I/O is fundamentally blocking
// tokio::fs uses thread pool (spawn_blocking)
// On Windows: file I/O can be async (IOCP)
// But tokio::fs still uses thread pool for consistency
// Linux has async I/O (io_uring) but it's complex
// tokio doesn't use it by default for file I/O
let _ = fs::read_to_string("data.txt").await?;
// Underneath, this is still blocking I/O on a thread pool
// It's not true async I/O like network operations
Ok(())
}File I/O is blocking on most platforms; tokio::fs hides this.
use memmap2::Mmap;
use std::fs::File;
#[tokio::main]
async fn main() -> std::io::Result<()> {
// For large files, memory mapping can be more efficient
// than either tokio::fs or std::fs
let file = tokio::task::spawn_blocking(|| {
let file = File::open("large_file.bin")?;
unsafe { Mmap::map(&file) }
}).await??;
// Now you have direct memory access
// No copying, no async overhead for reads
println!("Mapped {} bytes", file.len());
// Writing requires different mmap setup
// And careful handling of concurrent access
Ok(())
}Memory mapping can be more efficient for large files than either approach.
| Aspect | tokio::fs | std::fs |
|--------|-------------|-----------|
| API style | Async (.await) | Synchronous |
| Runtime impact | Non-blocking | Blocks executor |
| Per-operation overhead | Higher (spawn_blocking) | Lower |
| Thread pool | Uses blocking pool | Uses current thread |
| Concurrency | True concurrent operations | Serial on each thread |
| Best for | Servers, high-concurrency | CLI tools, batch, startup |
| File handles | tokio::fs::File | std::fs::File |
The choice between tokio::fs and std::fs centers on whether you're optimizing for throughput or for responsiveness:
tokio::fs prioritizes runtime responsiveness: File operations run on a separate blocking thread pool, allowing async tasks to continue making progress. This is essential in servers where blocking one task shouldn't block others. The cost is overhead: each operation requires scheduling work on the blocking pool, potentially waiting for a thread, and communicating results back. For many small operations, this overhead accumulates.
std::fs prioritizes raw throughput: Direct blocking calls have minimal overhead. In batch processing, CLI tools, or during application startup, blocking briefly is acceptable because there's no competing async work. The cost is that the async runtime cannot make progress during the operation—if you call std::fs::read_to_string in an async function, no other task on that thread can run until it completes.
Key insight: File I/O is fundamentally blocking on most operating systems. Unlike network sockets, which have mature async APIs (epoll, kqueue, IOCP), files typically require thread-based blocking. tokio::fs doesn't make file I/O async—it makes file I/O not block the async runtime. True async file I/O exists (Windows IOCP, Linux io_uring) but requires different APIs and more complexity.
Recommendation: Use tokio::fs for concurrent servers where responsiveness matters more than raw throughput. Use std::fs wrapped in manual spawn_blocking when you have multiple file operations to batch together, want explicit control over thread usage, or are writing non-async code. For startup, shutdown, or one-time operations, std::fs directly in async code is acceptable if the operation is fast—the brief block is often worth the simplicity.