Arc<Mutex>

Guides

Rust walkthroughs

← Back to all guides

What are the trade-offs between tokio::fs and std::fs for file operations in async code?

tokio::fs provides an async API for file operations that internally uses spawn_blocking to offload blocking I/O to a thread pool, preventing it from blocking the async runtime. std::fs performs synchronous blocking I/O on the current thread. The fundamental trade-off is that tokio::fs keeps the async runtime responsive by moving blocking work elsewhere, but incurs overhead from thread pool scheduling and context switches. std::fs is simpler and has lower overhead per operation but will block the async runtime, preventing other tasks from making progress. For high-concurrency async applications, tokio::fs is appropriate; for batch processing or when you control thread allocation explicitly, std::fs with manual spawn_blocking may be more efficient.

Blocking Nature of File I/O

use std::fs;
use std::io::Read;
 
fn main() -> std::io::Result<()> {
    // std::fs blocks the current thread
    let mut file = fs::File::open("data.txt")?;
    let mut contents = String::new();
    file.read_to_string(&mut contents)?;
    
    // This thread is blocked during the read
    // In async code, this blocks the entire executor
    
    println!("Read {} bytes", contents.len());
    Ok(())
}

std::fs operations are blocking at the OS level.

tokio::fs Async API

use tokio::fs;
use tokio::io::AsyncReadExt;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // tokio::fs provides async-style API
    let mut file = fs::File::open("data.txt").await?;
    let mut contents = String::new();
    file.read_to_string(&mut contents).await?;
    
    // The .await allows other tasks to run
    // Internally, this uses spawn_blocking
    
    println!("Read {} bytes", contents.len());
    Ok(())
}

tokio::fs provides .await points for async compatibility.

How tokio::fs Works Internally

use tokio::task::spawn_blocking;
use std::fs;
 
// tokio::fs::File::open is roughly equivalent to:
async fn tokio_file_open(path: &str) -> std::io::Result<tokio::fs::File> {
    let path = path.to_owned();
    spawn_blocking(move || {
        // This runs on the blocking thread pool
        fs::File::open(&path)
    }).await
}
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // tokio::fs operations use spawn_blocking internally
    // They offload to a separate thread pool
    
    let contents = tokio::fs::read_to_string("data.txt").await?;
    println!("Read {} bytes", contents.len());
    
    Ok(())
}

tokio::fs wraps blocking operations in spawn_blocking.

Runtime Blocking with std::fs

use std::fs;
use tokio::time::{sleep, Duration};
 
#[tokio::main]
async fn main() {
    // Spawn a task that should print periodically
    let ticker = tokio::spawn(async {
        let mut count = 0;
        loop {
            sleep(Duration::from_millis(100)).await;
            count += 1;
            println!("Tick {}", count);
        }
    });
    
    // This std::fs call blocks the runtime
    // The ticker task cannot run during this
    let _ = fs::read_to_string("large_file.txt");
    
    // Ticks are delayed while reading the file
    // The runtime cannot poll other tasks
    
    ticker.abort();
}

std::fs in async code blocks all tasks on the same thread.

Non-Blocking with tokio::fs

use tokio::fs;
use tokio::time::{sleep, Duration};
 
#[tokio::main]
async fn main() {
    // Spawn a task that prints periodically
    let ticker = tokio::spawn(async {
        let mut count = 0;
        loop {
            sleep(Duration::from_millis(100)).await;
            count += 1;
            println!("Tick {}", count);
        }
    });
    
    // tokio::fs yields during the operation
    // The ticker task continues running
    let _ = fs::read_to_string("large_file.txt").await;
    
    // Ticks continue while reading
    // The runtime remains responsive
    
    ticker.abort();
}

tokio::fs keeps the runtime responsive during file operations.

Overhead Comparison

use tokio::fs;
use std::fs;
use std::time::Instant;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    let iterations = 100;
    
    // std::fs: direct call, no async overhead
    let start = Instant::now();
    for _ in 0..iterations {
        let _ = fs::read_to_string("small.txt")?;
    }
    let std_time = start.elapsed();
    
    // tokio::fs: spawn_blocking overhead
    let start = Instant::now();
    for _ in 0..iterations {
        let _ = fs::read_to_string("small.txt").await?;
    }
    let tokio_time = start.elapsed();
    
    println!("std::fs: {:?}", std_time);
    println!("tokio::fs: {:?}", tokio_time);
    
    // For small files and many operations, std::fs is faster
    // The spawn_blocking overhead adds up
    
    Ok(())
}

std::fs has lower per-operation overhead than tokio::fs.

Manual spawn_blocking for Control

use tokio::task::spawn_blocking;
use std::fs;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // Instead of individual tokio::fs calls
    // Batch operations in one spawn_blocking
    
    let results = spawn_blocking(move || {
        // All blocking operations together
        let data1 = fs::read_to_string("file1.txt")?;
        let data2 = fs::read_to_string("file2.txt")?;
        let data3 = fs::read_to_string("file3.txt")?;
        
        Ok::<_, std::io::Error>((data1, data2, data3))
    }).await??;
    
    // Only one spawn_blocking call for multiple operations
    // Reduces thread pool scheduling overhead
    
    println!("Read {} bytes total", 
        results.0.len() + results.1.len() + results.2.len());
    
    Ok(())
}

Batching std::fs operations in spawn_blocking can be more efficient.

Thread Pool Configuration

use tokio::runtime::Builder;
 
#[tokio::main]
async fn main() {
    // Default: blocking thread pool has limited threads
    // Defaults to num_cpus * 2 for max_blocking_threads
}
 
fn custom_runtime() -> tokio::runtime::Runtime {
    Builder::new_multi_thread()
        .worker_threads(4)           // Async worker threads
        .max_blocking_threads(16)    // Blocking thread pool size
        .build()
        .unwrap()
}
 
// If you have many concurrent tokio::fs operations,
// you may need to increase max_blocking_threads
// Otherwise, operations queue up waiting for threads

Blocking thread pool size affects tokio::fs concurrency.

When tokio::fs Helps

use tokio::fs;
use tokio::net::TcpListener;
use tokio::io::AsyncWriteExt;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:8080").await?;
    
    loop {
        let (mut socket, _) = listener.accept().await?;
        
        // Handle each connection in a task
        tokio::spawn(async move {
            // Read file to send
            let content = match fs::read_to_string("response.txt").await {
                Ok(c) => c,
                Err(_) => return,
            };
            
            // Send to client
            let _ = socket.write_all(content.as_bytes()).await;
        });
    }
}

In servers, tokio::fs prevents one slow file read from blocking all connections.

When std::fs is Appropriate

use std::fs;
use tokio::task::spawn_blocking;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // For startup/shutdown operations
    // Where blocking briefly is acceptable
    
    let config = fs::read_to_string("config.json")?;
    println!("Loaded config");
    
    // For batch file processing
    // Where you want explicit control
    
    let results = spawn_blocking(|| {
        let mut total = 0;
        for entry in fs::read_dir("data/")? {
            let path = entry?.path();
            if path.extension().map(|e| e == "txt").unwrap_or(false) {
                let content = fs::read_to_string(&path)?;
                total += content.lines().count();
            }
        }
        Ok::<_, std::io::Error>(total)
    }).await??;
    
    println!("Total lines: {}", results);
    
    Ok(())
}

Use std::fs for startup, shutdown, or batch processing with explicit spawn_blocking.

Concurrent File Operations

use tokio::fs;
use tokio::task::JoinSet;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    let paths = vec!["file1.txt", "file2.txt", "file3.txt", "file4.txt"];
    
    let mut tasks = JoinSet::new();
    
    // Spawn concurrent reads
    for path in paths {
        tasks.spawn(async move {
            fs::read_to_string(path).await
        });
    }
    
    // Collect results as they complete
    let mut contents = Vec::new();
    while let Some(result) = tasks.join_next().await {
        contents.push(result??);
    }
    
    println!("Read {} files", contents.len());
    
    Ok(())
}

tokio::fs enables true concurrency for multiple file operations.

Error Handling Differences

use tokio::fs;
use std::fs;
 
#[tokio::main]
async fn main() {
    // std::fs: synchronous Result
    match std::fs::read_to_string("nonexistent.txt") {
        Ok(content) => println!("Content: {}", content),
        Err(e) => println!("Error: {}", e),
    }
    
    // tokio::fs: async Result (await gives io::Result)
    match fs::read_to_string("nonexistent.txt").await {
        Ok(content) => println!("Content: {}", content),
        Err(e) => println!("Error: {}", e),
    }
    
    // Error types are the same (std::io::Error)
    // tokio::fs wraps them in async machinery
}

Both use std::io::Error; tokio::fs adds async wrapping.

File Handle Differences

use tokio::fs::File;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use std::fs::File as StdFile;
use std::io::{Read, Write};
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // std::fs::File: blocking file handle
    let mut std_file = StdFile::open("data.txt")?;
    let mut buffer = vec![0u8; 1024];
    std_file.read(&mut buffer)?;
    // File handle is blocking
    
    // tokio::fs::File: async file handle
    let mut tokio_file = File::open("data.txt").await?;
    tokio_file.read(&mut buffer).await?;
    // File handle has async methods
    
    // Cannot use std::fs::File in async code without spawn_blocking
    // Cannot use tokio::fs::File in blocking code
    
    Ok(())
}

File handles are different types; use the matching API.

Platform-Specific Behavior

use tokio::fs;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // On Linux/macOS: file I/O is fundamentally blocking
    // tokio::fs uses thread pool (spawn_blocking)
    
    // On Windows: file I/O can be async (IOCP)
    // But tokio::fs still uses thread pool for consistency
    
    // Linux has async I/O (io_uring) but it's complex
    // tokio doesn't use it by default for file I/O
    
    let _ = fs::read_to_string("data.txt").await?;
    
    // Underneath, this is still blocking I/O on a thread pool
    // It's not true async I/O like network operations
    
    Ok(())
}

File I/O is blocking on most platforms; tokio::fs hides this.

Memory Mapping Alternative

use memmap2::Mmap;
use std::fs::File;
 
#[tokio::main]
async fn main() -> std::io::Result<()> {
    // For large files, memory mapping can be more efficient
    // than either tokio::fs or std::fs
    
    let file = tokio::task::spawn_blocking(|| {
        let file = File::open("large_file.bin")?;
        unsafe { Mmap::map(&file) }
    }).await??;
    
    // Now you have direct memory access
    // No copying, no async overhead for reads
    
    println!("Mapped {} bytes", file.len());
    
    // Writing requires different mmap setup
    // And careful handling of concurrent access
    
    Ok(())
}

Memory mapping can be more efficient for large files than either approach.

Summary Table

Aspect tokio::fs std::fs
API style Async (.await) Synchronous
Runtime impact Non-blocking Blocks executor
Per-operation overhead Higher (spawn_blocking) Lower
Thread pool Uses blocking pool Uses current thread
Concurrency True concurrent operations Serial on each thread
Best for Servers, high-concurrency CLI tools, batch, startup
File handles tokio::fs::File std::fs::File

Synthesis

The choice between tokio::fs and std::fs centers on whether you're optimizing for throughput or for responsiveness:

tokio::fs prioritizes runtime responsiveness: File operations run on a separate blocking thread pool, allowing async tasks to continue making progress. This is essential in servers where blocking one task shouldn't block others. The cost is overhead: each operation requires scheduling work on the blocking pool, potentially waiting for a thread, and communicating results back. For many small operations, this overhead accumulates.

std::fs prioritizes raw throughput: Direct blocking calls have minimal overhead. In batch processing, CLI tools, or during application startup, blocking briefly is acceptable because there's no competing async work. The cost is that the async runtime cannot make progress during the operation—if you call std::fs::read_to_string in an async function, no other task on that thread can run until it completes.

Key insight: File I/O is fundamentally blocking on most operating systems. Unlike network sockets, which have mature async APIs (epoll, kqueue, IOCP), files typically require thread-based blocking. tokio::fs doesn't make file I/O async—it makes file I/O not block the async runtime. True async file I/O exists (Windows IOCP, Linux io_uring) but requires different APIs and more complexity.

Recommendation: Use tokio::fs for concurrent servers where responsiveness matters more than raw throughput. Use std::fs wrapped in manual spawn_blocking when you have multiple file operations to batch together, want explicit control over thread usage, or are writing non-async code. For startup, shutdown, or one-time operations, std::fs directly in async code is acceptable if the operation is fast—the brief block is often worth the simplicity.