How does `bytes::BufMut` extend `Vec<u8>` for efficient buffer writing patterns?

BufMut is a trait that abstracts buffer writing operations, providing methods for writing various data types directly into a byte buffer without the overhead of individual bounds checks. While Vec<u8> requires manual reserve/resize management and forces you to convert types to bytes yourself, BufMut implementations like BytesMut track remaining capacity internally, provide specialized methods for each numeric type, and enable advanced patterns like zero-copy writes and efficient chunk management. The key efficiency gains come from: pre-validated capacity that eliminates per-write bounds checks, direct memory access for writing typed values, and the ability to work with both contiguous and non-contiguous underlying storage.

The Vec Baseline

fn vec_writing_baseline() {
    let mut buffer = Vec::new();
    
    // Manual reserve for capacity
    buffer.reserve(1024);
    
    // Writing a u32 requires manual conversion
    let value: u32 = 0x12345678;
    buffer.extend_from_slice(&value.to_be_bytes());
    
    // Writing multiple values
    for i in 0..100 {
        buffer.extend_from_slice(&(i as u32).to_be_bytes());
    }
    
    // Each extend_from_slice checks bounds
    // Each to_be_bytes() creates a temporary array
}

Writing to Vec<u8> requires manual capacity management and type-to-bytes conversion for each write.

Basic BufMut Usage

use bytes::BufMut;
 
fn bufmut_basic() {
    let mut buffer = bytes::BytesMut::new();
    
    // Reserve capacity upfront
    buffer.reserve(1024);
    
    // Write typed values directly
    buffer.put_u32(0x12345678);  // Big-endian by default
    buffer.put_u32_le(0x12345678);  // Little-endian
    
    // Write multiple values efficiently
    for i in 0..100 {
        buffer.put_u32(i);
    }
    
    // No intermediate byte arrays
    // No per-write bounds checks after initial reserve
}

BufMut provides typed methods that write directly into the buffer's memory.

Eliminating Bounds Checks

use bytes::BufMut;
 
fn bounds_check_elimination() {
    let mut buffer = bytes::BytesMut::with_capacity(1000);
    
    // Vec<u8> approach: each write checks capacity
    let mut vec = Vec::with_capacity(1000);
    for i in 0..100 {
        // Each extend_from_slice:
        // 1. Checks if there's enough capacity
        // 2. Potentially reallocates
        // 3. Copies bytes
        vec.extend_from_slice(&i.to_be_bytes());
    }
    
    // BufMut approach: chunk writes without checks
    let mut buf = bytes::BytesMut::with_capacity(1000);
    
    // Has_remaining() check once, then unchecked writes
    for i in 0..100 {
        buf.put_u64(i);
    }
    
    // When you know capacity is sufficient, use unchecked writes
    let mut buf = bytes::BytesMut::with_capacity(100);
    unsafe {
        // Caller guarantees there's enough space
        for i in 0..10 {
            buf.put_u64_unchecked(i);
        }
    }
}

BufMut can skip bounds checks when capacity is pre-validated, improving performance.

Typed Writing Methods

use bytes::BufMut;
 
fn typed_writing_methods() {
    let mut buf = bytes::BytesMut::with_capacity(256);
    
    // Integers - big endian (network byte order) by default
    buf.put_u8(0xFF);
    buf.put_i8(-128);
    buf.put_u16(0x1234);
    buf.put_i16(-1000);
    buf.put_u32(0x12345678);
    buf.put_i32(-100000);
    buf.put_u64(0x123456789ABCDEF0);
    buf.put_i64(-10000000000);
    buf.put_u128(0x123456789ABCDEF0123456789ABCDEF0);
    buf.put_i128(-100000000000000000000);
    
    // Little-endian variants
    buf.put_u16_le(0x1234);
    buf.put_u32_le(0x12345678);
    buf.put_u64_le(0x123456789ABCDEF0);
    
    // Floating point
    buf.put_f32(3.14159);
    buf.put_f64(3.14159265358979);
    buf.put_f32_le(3.14159);
    buf.put_f64_le(3.14159265358979);
    
    // Bytes and slices
    buf.put_slice(b"hello world");
    buf.put_bytes(0xAB, 10);  // Write 0xAB ten times
}

BufMut provides methods for every primitive type with explicit endianness.

BytesMut vs Vec Implementation

use bytes::{BufMut, BytesMut, Bytes};
 
fn bytesmut_features() {
    // BytesMut: the primary BufMut implementation
    let mut buf = BytesMut::with_capacity(1024);
    
    // Write some data
    buf.put_u32(0x12345678);
    buf.put_slice(b"hello");
    
    // Split the buffer - zero-copy extraction
    let written = buf.split();
    // written: BytesMut containing the written data
    // buf: Empty BytesMut ready for more writes
    
    println!("Written: {:?}", written);
    println!("Remaining buffer: {:?}", buf);
    
    // This enables efficient producer/consumer patterns
}
 
fn split_pattern() {
    // Common pattern for protocol implementations
    let mut buf = BytesMut::with_capacity(4096);
    
    loop {
        // Write data into buffer
        write_message(&mut buf);
        
        // Split off complete messages
        while buf.len() >= MESSAGE_HEADER_SIZE {
            let len = parse_message_length(&buf);
            if buf.len() < len {
                break;
            }
            
            // Zero-copy message extraction
            let message = buf.split_to(len);
            process_message(message);
        }
    }
}
 
fn write_message(buf: &mut BytesMut) {
    buf.put_u32(42);  // Message header
    buf.put_slice(b"payload");
}
 
fn parse_message_length(buf: &BytesMut) -> usize {
    4 + 7  // Header + payload
}
 
fn process_message(msg: BytesMut) {
    // Process without copying
}

BytesMut provides split() operations for zero-copy buffer management.

Advance and Remaining Capacity

use bytes::BufMut;
 
fn capacity_tracking() {
    let mut buf = bytes::BytesMut::with_capacity(100);
    
    // Check remaining capacity
    println!("Remaining: {}", buf.remaining_mut());
    
    // BufMut tracks remaining capacity internally
    // No need to manually check len() vs capacity()
    
    // Write data
    buf.put_slice(&[0u8; 50]);
    println!("Remaining after write: {}", buf.remaining_mut());
    
    // Advance: mark bytes as written without actually writing
    // Useful when you write directly to memory
    unsafe {
        // Get raw pointer to unwritten region
        let ptr = buf.chunk_mut().as_mut_ptr();
        
        // Write directly to memory (e.g., via FFI)
        // std::ptr::write_bytes(ptr, 0xAB, 20);
        
        // Tell BufMut we wrote 20 bytes
        buf.advance_mut(20);
    }
}
 
fn chunk_mut_pattern() {
    use bytes::BufMut;
    
    let mut buf = bytes::BytesMut::with_capacity(100);
    
    unsafe {
        // Access the unwritten portion directly
        let chunk = buf.chunk_mut();
        
        // chunk is a &mut [MaybeUninit<u8>]
        // You can write to it without initializing
        
        // After writing, advance the cursor
        buf.advance_mut(10);
    }
}

BufMut tracks remaining capacity and provides unsafe methods for direct memory access.

Resizing Behavior

use bytes::BufMut;
 
fn resizing_behavior() {
    let mut buf = bytes::BytesMut::new();
    println!("Initial capacity: {}", buf.capacity());
    
    // Automatic resizing on write
    buf.put_u64(42);
    println!("After u64: capacity {}", buf.capacity());
    
    // Writing more than capacity triggers resize
    buf.put_slice(&[0u8; 1000]);
    println!("After 1000 bytes: capacity {}", buf.capacity());
    
    // With reserve, you control when allocation happens
    let mut buf2 = bytes::BytesMut::new();
    buf2.reserve(10000);
    println!("After reserve: capacity {}", buf2.capacity());
    
    // Now writes don't trigger reallocation
    buf2.put_slice(&[0u8; 5000]);
    println!("After 5000 bytes: capacity {}", buf2.capacity());
}

BytesMut automatically resizes like Vec, but you can pre-reserve capacity.

Zero-Copy Patterns

use bytes::{BufMut, BytesMut, Bytes};
 
fn zero_copy_patterns() {
    let mut buf = BytesMut::with_capacity(1024);
    
    // Write some data
    buf.put_slice(b"Hello, ");
    buf.put_slice(b"World!");
    
    // split() returns the written portion without copying
    let data: Bytes = buf.split().freeze();
    // data references the original allocation
    // No copy was made!
    
    // freeze() converts BytesMut to Bytes (immutable, shareable)
    
    // Multiple readers can share the same Bytes
    let reader1 = data.clone();
    let reader2 = data.clone();
    // Still no copies - reference counted
    
    // The original buf can be reused
    buf.put_slice(b"Next message");
}
 
fn efficient_concatenation() {
    let mut buf = BytesMut::with_capacity(100);
    
    buf.put_slice(b"Header: ");
    buf.put_u32(0x12345678);
    buf.put_slice(b" Data: ");
    buf.put_slice(b"payload content here");
    
    // All written contiguously in one buffer
    // No intermediate allocations
    
    let message = buf.split().freeze();
    // message is a single contiguous Bytes
}

BytesMut::split() and freeze() enable zero-copy data extraction and sharing.

Comparison with Vec Write Patterns

use bytes::BufMut;
 
fn vec_pattern() {
    // Vec<u8> pattern
    let mut vec = Vec::with_capacity(100);
    
    // Write header
    vec.extend_from_slice(b"HEAD");
    
    // Write length (must convert to bytes)
    let len: u32 = 42;
    vec.extend_from_slice(&len.to_be_bytes());
    
    // Write payload
    vec.extend_from_slice(b"payload");
    
    // Converting to shared reference requires clone or Arc
    let shared: Vec<u8> = vec.clone();  // Full copy!
}
 
fn bufmut_pattern() {
    // BufMut pattern
    let mut buf = bytes::BytesMut::with_capacity(100);
    
    // Write header
    buf.put_slice(b"HEAD");
    
    // Write length (no intermediate array)
    buf.put_u32(42);
    
    // Write payload
    buf.put_slice(b"payload");
    
    // Convert to shared reference without copy
    let shared: bytes::Bytes = buf.split().freeze();  // Zero copy!
}

BufMut avoids intermediate allocations and enables zero-copy sharing.

Implementing BufMut for Custom Types

use bytes::BufMut;
 
// You can implement BufMut for your own buffer types
struct LogBuffer {
    data: Vec<u8>,
    written: usize,
}
 
impl BufMut for LogBuffer {
    fn remaining_mut(&self) -> usize {
        self.data.capacity() - self.written
    }
    
    unsafe fn advance_mut(&mut self, cnt: usize) {
        self.written += cnt;
    }
    
    fn chunk_mut(&mut self) -> &mut bytes::buf::UninitSlice {
        // Safety: we're returning a mutable slice to unwritten memory
        unsafe {
            let ptr = self.data.as_mut_ptr().add(self.written);
            let len = self.data.capacity() - self.written;
            bytes::buf::UninitSlice::from_raw_parts_mut(ptr, len)
        }
    }
}
 
impl LogBuffer {
    fn new(capacity: usize) -> Self {
        Self {
            data: vec![0; capacity],
            written: 0,
        }
    }
    
    fn as_slice(&self) -> &[u8] {
        &self.data[..self.written]
    }
}
 
fn custom_bufmut() {
    let mut log = LogBuffer::new(100);
    log.put_u32(0x12345678);
    log.put_slice(b"log entry");
    
    println!("Written: {:?}", log.as_slice());
}

BufMut can be implemented for custom buffer types with specific requirements.

Working with IoBufMut

use bytes::BufMut;
 
// For async I/O, BytesMut implements IoBufMut (with tokio feature)
#[cfg(feature = "tokio")]
fn async_io_pattern() {
    use bytes::BytesMut;
    use tokio::io::AsyncWriteExt;
    
    async fn write_message(mut writer: impl AsyncWriteExt + Unpin) {
        let mut buf = BytesMut::with_capacity(1024);
        
        // Build message in buffer
        buf.put_u32(42);  // Header
        buf.put_slice(b"Hello, async world!");
        
        // Write directly from buffer
        // writer.write_buf(&mut buf).await.unwrap();
    }
}

BytesMut integrates with async I/O for efficient buffer management.

Real-World Example: Protocol Implementation

use bytes::{BufMut, BytesMut};
 
struct Message {
    msg_type: u8,
    flags: u8,
    payload: Vec<u8>,
}
 
impl Message {
    fn encode(&self, buf: &mut BytesMut) {
        // Reserve space for header + length + payload
        buf.reserve(4 + 4 + self.payload.len());
        
        // Write header
        buf.put_u8(self.msg_type);
        buf.put_u8(self.flags);
        buf.put_u16_be(self.payload.len() as u16);
        
        // Write payload
        buf.put_slice(&self.payload);
    }
    
    fn decode(buf: &mut BytesMut) -> Option<Self> {
        // Check if we have enough bytes for header
        if buf.len() < 4 {
            return None;
        }
        
        // Peek at length (don't consume)
        let len = u16::from_be_bytes([buf[2], buf[3]]) as usize;
        
        // Check if we have full message
        if buf.len() < 4 + len {
            return None;
        }
        
        // Now consume
        let msg_type = buf.get_u8();
        let flags = buf.get_u8();
        let _len = buf.get_u16();
        let payload = buf.split_to(len).to_vec();
        
        Some(Self { msg_type, flags, payload })
    }
}
 
fn protocol_pattern() {
    let mut buf = BytesMut::new();
    
    // Encode messages
    let msg1 = Message {
        msg_type: 1,
        flags: 0,
        payload: b"hello".to_vec(),
    };
    msg1.encode(&mut buf);
    
    let msg2 = Message {
        msg_type: 2,
        flags: 0,
        payload: b"world".to_vec(),
    };
    msg2.encode(&mut buf);
    
    // Decode messages
    while let Some(msg) = Message::decode(&mut buf) {
        println!("Message type: {}, payload: {:?}", msg.msg_type, msg.payload);
    }
}

BytesMut with BufMut methods provides a natural API for protocol encoding/decoding.

Memory Layout and Performance

use bytes::BufMut;
 
fn memory_efficiency() {
    // Vec<u8> approach with intermediate allocations
    let mut vec = Vec::new();
    
    // Each to_be_bytes() creates a temporary [u8; 8]
    for i in 0..1000 {
        vec.extend_from_slice(&(i as u64).to_be_bytes());
    }
    // 1000 temporary [u8; 8] arrays created
    
    // BufMut approach: direct write
    let mut buf = bytes::BytesMut::with_capacity(8000);
    for i in 0..1000 {
        buf.put_u64(i);
    }
    // No intermediate arrays
    // Direct memory write
    
    // Vec approach after reserve
    let mut vec2 = Vec::with_capacity(8000);
    for i in 0..1000 {
        vec2.extend_from_slice(&(i as u64).to_be_bytes());
    }
    // Still has intermediate arrays, but no reallocations
}
 
fn benchmark_comparison() {
    use std::time::Instant;
    
    let iterations = 100_000;
    
    // Vec<u8>
    let start = Instant::now();
    let mut vec = Vec::with_capacity(iterations * 8);
    for i in 0..iterations {
        vec.extend_from_slice(&(i as u64).to_be_bytes());
    }
    let vec_time = start.elapsed();
    
    // BytesMut
    let start = Instant::now();
    let mut buf = bytes::BytesMut::with_capacity(iterations * 8);
    for i in 0..iterations {
        buf.put_u64(i);
    }
    let buf_time = start.elapsed();
    
    println!("Vec<u8>: {:?}", vec_time);
    println!("BytesMut: {:?}", buf_time);
    // BytesMut is typically faster due to:
    // 1. No intermediate byte arrays
    // 2. Fewer bounds checks
    // 3. Direct memory writes
}

BufMut eliminates intermediate allocations and reduces bounds checking overhead.

Integration with Other Crates

use bytes::BytesMut;
use bytes::BufMut;
 
fn tokio_integration() {
    // BytesMut works well with Tokio's async I/O
    // (requires tokio feature in bytes crate)
    
    // let mut buf = BytesMut::with_capacity(4096);
    // 
    // // Read into buffer
    // let n = async_stdin.read_buf(&mut buf).await?;
    // 
    // // Process data
    // while buf.has_remaining() {
    //     process_byte(buf.get_u8());
    // }
}
 
fn prost_integration() {
    // Protocol buffers with bytes::BufMut
    // prost uses BufMut for encoding
    
    // let mut buf = BytesMut::new();
    // my_message.encode(&mut buf).unwrap();
    // 
    // // buf now contains the encoded protobuf
    // let bytes = buf.freeze();
}

BytesMut integrates seamlessly with the Rust async ecosystem.

Comparison Summary

| Feature | Vec<u8> | BufMut (BytesMut) | |---------|-----------|---------------------| | Typed writes | Manual (to_be_bytes()) | Built-in (put_u32, etc.) | | Bounds checking | Every extend_from_slice | Once per operation | | Zero-copy extraction | Clone required | split() + freeze() | | Shared ownership | Arc<Vec<u8>> or clone | Bytes (reference counted) | | Endianness | Manual | Built-in (_le, _be suffixes) | | Direct memory access | as_mut_ptr() + unsafe | chunk_mut() + advance_mut() | | Capacity tracking | len() vs capacity() | remaining_mut() | | Async I/O integration | Manual buffer management | read_buf, write_buf |

Synthesis

BufMut extends the basic Vec<u8> with efficient, typed, and zero-copy buffer writing capabilities:

Key advantages over Vec<u8>:

Typed writes: put_u32, put_f64, etc. write directly without intermediate byte arrays
Reduced bounds checking: Pre-validated capacity enables unchecked writes
Zero-copy extraction: split() and freeze() create shareable references without copying
Endianness control: Explicit put_u32_be and put_u32_le methods
Direct memory access: chunk_mut() and advance_mut() for FFI and custom writes

When to use BufMut:

Network protocol implementation (encoding/decoding messages)
Binary serialization (when serde isn't appropriate)
High-performance I/O (eliminating copies in the hot path)
Producer/consumer patterns with shared buffers
Any situation where you're building byte buffers incrementally

When Vec is sufficient:

Simple, infrequent byte buffer construction
When you don't need zero-copy sharing
When intermediate allocations don't matter
When you're not writing typed values

Key insight: BufMut transforms buffer writing from a sequence of "convert to bytes then copy" operations into direct typed writes, eliminating both the intermediate byte arrays and the redundant bounds checks. Combined with BytesMut's split() functionality, this enables patterns where data is written once and then shared across multiple readers without any copies—a critical capability for high-performance network servers and protocol implementations.

Loading page…

How does bytes::BufMut extend Vec<u8> for efficient buffer writing patterns?

The Vec Baseline

Basic BufMut Usage

Eliminating Bounds Checks

Typed Writing Methods

BytesMut vs Vec Implementation

Advance and Remaining Capacity

Resizing Behavior

Zero-Copy Patterns

Comparison with Vec Write Patterns

Implementing BufMut for Custom Types

Working with IoBufMut

Real-World Example: Protocol Implementation

Memory Layout and Performance

Integration with Other Crates

Comparison Summary

Synthesis

How does bytes::BufMut extend Vec<u8> for efficient buffer writing patterns?

The Vec Baseline

Basic BufMut Usage

Eliminating Bounds Checks

Typed Writing Methods

BytesMut vs Vec Implementation

Advance and Remaining Capacity

Resizing Behavior

Zero-Copy Patterns

Comparison with Vec Write Patterns

Implementing BufMut for Custom Types

Working with IoBufMut

Real-World Example: Protocol Implementation

Memory Layout and Performance

Integration with Other Crates

Comparison Summary

Synthesis

How does `bytes::BufMut` extend `Vec<u8>` for efficient buffer writing patterns?

How does `bytes::BufMut` extend `Vec<u8>` for efficient buffer writing patterns?