What are the differences between `zstd::Encoder` and `zstd::Decoder` streaming interfaces for compression workflows?

The zstd crate provides streaming interfaces for both compression and decompression through Encoder and Decoder types. While they share similar patterns, their interfaces have important differences that affect how you structure compression workflows, handle completion, and manage resources.

Basic Encoder Usage

use std::fs::File;
use std::io::{self, BufWriter, Read, Write};
use zstd::Encoder;
 
fn compress_file(input_path: &str, output_path: &str) -> io::Result<()> {
    let input = File::open(input_path)?;
    let output = File::create(output_path)?;
    
    // Encoder wraps the output writer
    let mut encoder = Encoder::new(output, 3)?;
    // Compression level 3 (default, good balance)
    
    // Read input and write to encoder
    let mut reader = input;
    io::copy(&mut reader, &mut encoder)?;
    
    // Important: must call finish() to complete compression
    encoder.finish()?;
    
    Ok(())
}

The encoder wraps an output writer and must be explicitly finished.

Basic Decoder Usage

use std::fs::File;
use std::io;
use zstd::Decoder;
 
fn decompress_file(input_path: &str, output_path: &str) -> io::Result<()> {
    let input = File::open(input_path)?;
    let output = File::create(output_path)?;
    
    // Decoder wraps the input reader
    let mut decoder = Decoder::new(input)?;
    
    // Read from decoder, write to output
    io::copy(&mut decoder, &output)?;
    
    // No explicit finish needed - decoder completes on drop
    // (all data read when io::copy finishes)
    
    Ok(())
}

The decoder wraps an input reader and completes automatically.

The Key Interface Difference: Completion

use std::io::{self, Cursor};
use zstd::{Encoder, Decoder};
 
fn completion_difference() -> io::Result<()> {
    // ENCODER: Must call finish()
    let buffer = Vec::new();
    let mut encoder = Encoder::new(buffer, 3)?;
    encoder.write_all(b"Hello, World!")?;
    
    // WRONG: Just dropping loses data
    // drop(encoder);  // Data is lost!
    
    // CORRECT: Call finish() to get compressed output
    let compressed = encoder.finish()?;
    println!("Compressed {} bytes", compressed.len());
    
    // DECODER: No finish() needed
    let cursor = Cursor::new(compressed);
    let mut decoder = Decoder::new(cursor)?;
    
    let mut decompressed = Vec::new();
    io::copy(&mut decoder, &mut decompressed)?;
    
    // Decoder completes automatically when input is exhausted
    println!("Decompressed: {}", String::from_utf8_lossy(&decompressed));
    
    Ok(())
}

Encoder requires finish(); decoder does not.

Encoder Type Signature

use std::io::{self, Write};
use zstd::Encoder;
 
fn encoder_signature() -> io::Result<()> {
    // Encoder<W> where W: Write
    // Creates compressed output
    
    let output: Vec<u8> = Vec::new();
    let encoder: Encoder<Vec<u8>> = Encoder::new(output, 3)?;
    
    // Encoder implements Write
    // Writing to encoder compresses data
    // Compressed data goes to inner writer
    
    // finish() returns the inner writer
    let mut encoder = Encoder::new(Vec::new(), 3)?;
    encoder.write_all(b"data")?;
    let compressed: Vec<u8> = encoder.finish()?;
    
    // The compressed Vec<u8> contains the complete zstd frame
    Ok(())
}

Encoder<W> wraps a Write type and returns it on finish().

Decoder Type Signature

use std::io::{self, Cursor, Read};
use zstd::Decoder;
 
fn decoder_signature() -> io::Result<()> {
    // Decoder<R> where R: BufRead
    // Note: Requires BufRead, not just Read
    
    let compressed = vec![/* zstd data */];
    let cursor = Cursor::new(compressed);
    
    // Decoder::new() requires BufRead
    let decoder: Decoder<Cursor<Vec<u8>>> = Decoder::new(cursor)?;
    
    // Decoder implements Read
    // Reading from decoder decompresses data
    
    let mut decoder = Decoder::new(Cursor::new(vec![]))?;
    let mut output = Vec::new();
    decoder.read_to_end(&mut output)?;
    
    // No finish() - decoder is just a Read adapter
    
    Ok(())
}

Decoder<R> wraps a BufRead type and implements Read.

Compression Levels

use std::io::{self, Cursor};
use zstd::Encoder;
 
fn compression_levels() -> io::Result<()> {
    let data = b"Hello, World! This is a test of compression levels.";
    
    // Level 0: Default (usually level 3)
    let mut encoder = Encoder::new(Vec::new(), 0)?;
    encoder.write_all(data)?;
    let level_0 = encoder.finish()?;
    
    // Level 3: Default, good balance
    let mut encoder = Encoder::new(Vec::new(), 3)?;
    encoder.write_all(data)?;
    let level_3 = encoder.finish()?;
    
    // Level 19: Maximum compression, slower
    let mut encoder = Encoder::new(Vec::new(), 19)?;
    encoder.write_all(data)?;
    let level_19 = encoder.finish()?;
    
    // Level 1: Fast compression, larger output
    let mut encoder = Encoder::new(Vec::new(), 1)?;
    encoder.write_all(data)?;
    let level_1 = encoder.finish()?;
    
    println!("Level 1:  {} bytes", level_1.len());
    println!("Level 3:  {} bytes", level_3.len());
    println!("Level 19: {} bytes", level_19.len());
    
    Ok(())
}

Encoder accepts compression level; decoder does not need it.

Streaming Large Files

use std::fs::File;
use std::io::{self, BufReader, BufWriter};
use zstd::{Encoder, Decoder};
 
fn stream_compress(large_input: &str, output: &str) -> io::Result<()> {
    let input = File::open(large_input)?;
    let output_file = File::create(output)?;
    
    // Use buffered I/O for performance
    let reader = BufReader::new(input);
    let writer = BufWriter::new(output_file);
    
    // Encoder handles streaming internally
    let mut encoder = Encoder::new(writer, 3)?;
    io::copy(&mut reader.borrow_mut(), &mut encoder)?;
    
    // finish() writes final zstd frame data
    encoder.finish()?;
    
    Ok(())
}
 
fn stream_decompress(compressed: &str, output: &str) -> io::Result<()> {
    let input = File::open(compressed)?;
    let output_file = File::create(output)?;
    
    // Decoder requires BufRead
    let reader = BufReader::new(input);
    let mut writer = BufWriter::new(output_file);
    
    let mut decoder = Decoder::new(reader)?;
    io::copy(&mut decoder, &mut writer)?;
    
    // Flush writer to ensure all data is written
    writer.flush()?;
    
    Ok(())
}

Both support streaming with constant memory usage for large files.

Chaining Operations

use std::io::{self, Cursor, Read, Write};
use zstd::{Encoder, Decoder};
 
fn chain_operations() -> io::Result<()> {
    let original = b"Data to compress";
    
    // Compress
    let mut encoder = Encoder::new(Vec::new(), 3)?;
    encoder.write_all(original)?;
    let compressed = encoder.finish()?;
    
    // Decompress
    let cursor = Cursor::new(compressed);
    let mut decoder = Decoder::new(cursor)?;
    let mut decompressed = Vec::new();
    decoder.read_to_end(&mut decompressed)?;
    
    assert_eq!(original.to_vec(), decompressed);
    
    // Chain: compress to memory, then decompress
    let compress = |data: &[u8]| -> io::Result<Vec<u8>> {
        let mut encoder = Encoder::new(Vec::new(), 3)?;
        encoder.write_all(data)?;
        encoder.finish()
    };
    
    let decompress = |data: Vec<u8>| -> io::Result<Vec<u8>> {
        let mut decoder = Decoder::new(Cursor::new(data))?;
        let mut output = Vec::new();
        decoder.read_to_end(&mut output)?;
        Ok(output)
    };
    
    let c = compress(b"test")?;
    let d = decompress(c)?;
    assert_eq!(d, b"test");
    
    Ok(())
}

The finish/return pattern enables chaining operations.

Error Handling Differences

use std::io::{self, Cursor};
use zstd::{Encoder, Decoder};
 
fn encoder_errors() -> io::Result<()> {
    // Encoder errors during write or finish
    let mut encoder = Encoder::new(Vec::new(), 3)?;
    
    // Write errors propagate
    encoder.write_all(b"data")?;
    
    // finish() can fail if compression fails
    let compressed = encoder.finish()?;
    
    Ok(())
}
 
fn decoder_errors() -> io::Result<()> {
    // Decoder errors during read
    let invalid_data = vec![0, 1, 2, 3];  // Not valid zstd
    let cursor = Cursor::new(invalid_data);
    let mut decoder = Decoder::new(cursor)?;
    
    let mut output = Vec::new();
    
    // read_to_end will fail on invalid zstd data
    match decoder.read_to_end(&mut output) {
        Err(e) => println!("Decompression failed: {}", e),
        Ok(_) => println!("Success"),
    }
    
    Ok(())
}

Encoder errors occur on write/finish; decoder errors occur on read.

Working with Buffers

use std::io::{self, Cursor, Read, Write};
use zstd::{Encoder, Decoder};
 
fn buffer_operations() -> io::Result<()> {
    // Encoding to a buffer
    fn compress_to_buffer(data: &[u8]) -> io::Result<Vec<u8>> {
        let mut encoder = Encoder::new(Vec::new(), 3)?;
        encoder.write_all(data)?;
        encoder.finish()
    }
    
    // Decoding from a buffer
    fn decompress_from_buffer(compressed: &[u8]) -> io::Result<Vec<u8>> {
        let cursor = Cursor::new(compressed);
        let mut decoder = Decoder::new(cursor)?;
        let mut output = Vec::new();
        decoder.read_to_end(&mut output)?;
        Ok(output)
    }
    
    // Usage
    let original = b"Hello, buffer!";
    let compressed = compress_to_buffer(original)?;
    let decompressed = decompress_from_buffer(&compressed)?;
    
    assert_eq!(original.to_vec(), decompressed);
    
    Ok(())
}

Both work naturally with in-memory buffers.

Multiple Frames

use std::io::{self, Cursor, Read, Write};
use zstd::{Encoder, Decoder};
 
fn multiple_frames() -> io::Result<()> {
    // zstd supports concatenated frames
    let mut combined = Vec::new();
    
    // Create first frame
    let mut encoder1 = Encoder::new(Vec::new(), 3)?;
    encoder1.write_all(b"Frame 1")?;
    combined.extend(encoder1.finish()?);
    
    // Create second frame
    let mut encoder2 = Encoder::new(Vec::new(), 3)?;
    encoder2.write_all(b"Frame 2")?;
    combined.extend(encoder2.finish()?);
    
    // Decoder reads all frames by default
    let cursor = Cursor::new(&combined);
    let mut decoder = Decoder::new(cursor)?;
    
    let mut all_data = Vec::new();
    decoder.read_to_end(&mut all_data)?;
    
    // Contains data from both frames
    assert_eq!(all_data, b"Frame 1Frame 2");
    
    Ok(())
}

Decoder handles multiple concatenated zstd frames automatically.

Encoder Configuration Options

use std::io;
use zstd::Encoder;
 
fn encoder_options() -> io::Result<()> {
    let mut encoder = Encoder::new(Vec::new(), 3)?;
    
    // Set dictionary for compression
    // encoder.set_dictionary(&dict)?;
    
    // Set number of threads for parallel compression
    encoder.multithread(4)?;  // Use 4 threads
    
    // Set window log (affects memory usage)
    // encoder.set_window_log(20)?;  // 2^20 window
    
    encoder.write_all(b"data")?;
    let compressed = encoder.finish()?;
    
    Ok(())
}

Encoder has configuration options; decoder generally does not need them.

Decoder Options

use std::io::{self, Cursor};
use zstd::Decoder;
 
fn decoder_options() -> io::Result<()> {
    let compressed = vec![/* zstd data */];
    let cursor = Cursor::new(compressed);
    
    // Single-frame mode (error on multiple frames)
    let decoder = Decoder::with_buffer(cursor)?;
    
    // Or use new() for multi-frame mode (default)
    let cursor2 = Cursor::new(vec![/* ... */]);
    let decoder2 = Decoder::new(cursor2)?;
    
    Ok(())
}

Decoder options are mainly about frame handling.

Manual vs Automatic Completion

use std::io::{self, Write};
use zstd::Encoder;
 
fn manual_completion() -> io::Result<()> {
    let mut encoder = Encoder::new(Vec::new(), 3)?;
    encoder.write_all(b"data")?;
    
    // Method 1: finish() returns inner writer
    let compressed = encoder.finish()?;
    
    // Method 2: Use into_inner() if you want to access writer
    // without completing compression (WARNING: may have partial data)
    let mut encoder2 = Encoder::new(Vec::new(), 3)?;
    encoder2.write_all(b"data")?;
    // let (writer, result) = encoder2.into_inner()?;  // Rarely needed
    
    Ok(())
}
 
fn automatic_completion() {
    // Decoder needs no explicit completion
    use std::io::Cursor;
    use zstd::Decoder;
    
    let compressed = vec![/* ... */];
    let cursor = Cursor::new(compressed);
    
    // When dropped, decoder just releases resources
    {
        let decoder = Decoder::new(cursor).unwrap();
        // read data...
    }  // Automatically cleaned up
}

Encoder completion is explicit; decoder cleanup is automatic.

Integration with Other Streams

use std::fs::File;
use std::io::{self, BufReader, BufWriter, Read, Write};
use zstd::{Encoder, Decoder};
 
fn compress_with_progress(input_path: &str, output_path: &str) -> io::Result<()> {
    let input = File::open(input_path)?;
    let output = File::create(output_path)?;
    
    let mut encoder = Encoder::new(BufWriter::new(output), 3)?;
    let mut reader = BufReader::new(input);
    
    // Manual copy with progress
    let mut buffer = [0u8; 8192];
    let mut total = 0;
    
    loop {
        let n = reader.read(&mut buffer)?;
        if n == 0 {
            break;
        }
        encoder.write_all(&buffer[..n])?;
        total += n;
        println!("Compressed {} bytes", total);
    }
    
    let _ = encoder.finish()?;
    println!("Done");
    
    Ok(())
}
 
fn decompress_with_validation(input_path: &str) -> io::Result<Vec<u8>> {
    let input = File::open(input_path)?;
    let reader = BufReader::new(input);
    
    let mut decoder = Decoder::new(reader)?;
    
    let mut output = Vec::new();
    let mut buffer = [0u8; 8192];
    
    loop {
        let n = decoder.read(&mut buffer)?;
        if n == 0 {
            break;
        }
        output.extend_from_slice(&buffer[..n]);
    }
    
    Ok(output)
}

Both integrate with the standard Read/Write traits.

Common Patterns Comparison

use std::io::{self, Cursor, Read, Write};
use zstd::{Encoder, Decoder};
 
fn patterns_comparison() -> io::Result<()> {
    // PATTERN: Compress data
    fn compress(data: &[u8]) -> io::Result<Vec<u8>> {
        let mut encoder = Encoder::new(Vec::new(), 3)?;
        encoder.write_all(data)?;
        encoder.finish()  // Returns compressed Vec<u8>
    }
    
    // PATTERN: Decompress data
    fn decompress(compressed: &[u8]) -> io::Result<Vec<u8>> {
        let cursor = Cursor::new(compressed);
        let mut decoder = Decoder::new(cursor)?;
        let mut output = Vec::new();
        decoder.read_to_end(&mut output)?;  // Reads all decompressed data
        Ok(output)
    }
    
    // PATTERN: Compress to writer
    fn compress_to<W: Write>(data: &[u8], writer: W) -> io::Result<W> {
        let mut encoder = Encoder::new(writer, 3)?;
        encoder.write_all(data)?;
        encoder.finish()
    }
    
    // PATTERN: Decompress from reader
    fn decompress_from<R: Read>(reader: R) -> io::Result<Vec<u8>> {
        // Note: Decoder needs BufRead, so wrap if needed
        let reader = io::BufReader::new(reader);
        let mut decoder = Decoder::new(reader)?;
        let mut output = Vec::new();
        decoder.read_to_end(&mut output)?;
        Ok(output)
    }
    
    Ok(())
}

The patterns differ mainly in completion handling.

Synthesis

The zstd::Encoder and Decoder streaming interfaces have complementary but asymmetric designs:

Type relationships:

| Type | Wraps | Implements | Returns on completion | |------|-------|------------|----------------------| | Encoder<W> | W: Write | Write | W via finish() | | Decoder<R> | R: BufRead | Read | Nothing (auto-complete) |

Key differences:

| Aspect | Encoder | Decoder | |--------|---------|---------| | Direction | Write → compress | Read ← decompress | | Trait bound | W: Write | R: BufRead | | Completion | finish() required | Automatic | | Output | Returns inner writer | Just reads | | Configuration | Level, threads, dictionary | Frame handling | | Error timing | On write or finish | On read |

Common patterns:

// Encode pattern
let mut encoder = Encoder::new(writer, level)?;
encoder.write_all(data)?;
let writer = encoder.finish()?;
 
// Decode pattern
let mut decoder = Decoder::new(reader)?;
decoder.read_to_end(&mut output)?;
// No finish needed

Memory characteristics:

| Operation | Memory usage | |-----------|--------------| | Encoding small data | Compressed size + buffers | | Encoding large files | Constant (streaming) | | Decoding small data | Decompressed size | | Decoding large files | Constant (streaming) |

Use streaming interfaces when dealing with data larger than memory or when integrating with other stream-based APIs.

Loading page…

What are the differences between zstd::Encoder and zstd::Decoder streaming interfaces for compression workflows?

Basic Encoder Usage

Basic Decoder Usage

The Key Interface Difference: Completion

Encoder Type Signature

Decoder Type Signature

Compression Levels

Streaming Large Files

Chaining Operations

Error Handling Differences

Working with Buffers

Multiple Frames

Encoder Configuration Options

Decoder Options

Manual vs Automatic Completion

Integration with Other Streams

Common Patterns Comparison

Synthesis

What are the differences between zstd::Encoder and zstd::Decoder streaming interfaces for compression workflows?

Basic Encoder Usage

Basic Decoder Usage

The Key Interface Difference: Completion

Encoder Type Signature

Decoder Type Signature

Compression Levels

Streaming Large Files

Chaining Operations

Error Handling Differences

Working with Buffers

Multiple Frames

Encoder Configuration Options

Decoder Options

Manual vs Automatic Completion

Integration with Other Streams

Common Patterns Comparison

Synthesis

What are the differences between `zstd::Encoder` and `zstd::Decoder` streaming interfaces for compression workflows?

What are the differences between `zstd::Encoder` and `zstd::Decoder` streaming interfaces for compression workflows?