How does bytes::Buf::chain enable zero-copy concatenation of multiple buffers?
chain creates a virtual concatenation of buffers by returning a Chain type that presents multiple underlying buffers as a single contiguous view without copying data. When you call buf1.chain(buf2), you get a new buffer that logically contains all bytes from buf1 followed by all bytes from buf2, but the data remains in its original locations. The Chain type implements the Buf trait, providing methods like copy_to_bytes() for when you actually need contiguous data, while operations like iteration and slicing work across the chained buffers transparently. This approach avoids the allocation and copying overhead of traditional concatenation, especially valuable in network code where data arrives in chunks but needs to be processed as a unit.
Basic chain Usage
use bytes::{Bytes, Buf};
fn main() {
let buf1 = Bytes::from("Hello, ");
let buf2 = Bytes::from("World!");
// chain creates a view over both buffers
let chained = buf1.chain(buf2);
// The chained buffer has combined length
assert_eq!(chained.remaining(), 13);
// Read as if it were a single buffer
let mut result = vec![0u8; 13];
chained.copy_to_slice(&mut result);
assert_eq!(&result, b"Hello, World!");
}chain creates a logical view without copying the underlying data.
Zero-Copy Demonstration
use bytes::{Bytes, Buf};
fn main() {
let original = Bytes::from("This is original data");
// Split without copying
let (part1, part2) = original.split_at(10);
// Chain back together - still no copy
let rejoined = part1.chain(part2);
// All views reference the same underlying storage
// No allocations or copies occurred
// Only copy when needed
let copied = rejoined.copy_to_bytes(rejoined.remaining());
println!("Original reference count stays shared");
}The underlying Bytes storage uses reference counting, allowing zero-cost slicing and chaining.
Chain with Multiple Buffers
use bytes::{Bytes, Buf};
fn main() {
let buffers: Vec<Bytes> = vec![
Bytes::from("First "),
Bytes::from("Second "),
Bytes::from("Third "),
];
// Chain all buffers together
let chained = buffers.into_iter()
.fold(Bytes::new().chain(Bytes::new()), |acc, buf| {
acc.chain(buf)
});
// Or use chain's collect pattern (more idiomatic)
let buffers2: Vec<Bytes> = vec![
Bytes::from("A"),
Bytes::from("B"),
Bytes::from("C"),
];
// Start with empty buffer and chain each
let mut chained = Bytes::new().chain(Bytes::new());
for buf in buffers2 {
chained = chained.chain(buf);
}
}Multiple buffers can be chained together, though there are more efficient patterns.
Iterating Over Chained Buffers
use bytes::{Bytes, Buf};
fn main() {
let buf1 = Bytes::from("ABC");
let buf2 = Bytes::from("DEF");
let chained = buf1.chain(buf2);
// Iterate over chunks
// Buf::chunk returns the next contiguous chunk
let mut total = 0;
let mut chunks = vec![];
let mut cursor = chained;
while cursor.has_remaining() {
let chunk = cursor.chunk();
chunks.push(std::str::from_utf8(chunk).unwrap());
total += chunk.len();
cursor.advance(chunk.len());
}
// The Chain yields chunks from each underlying buffer
// You see "ABC" then "DEF" as separate chunks
assert_eq!(chunks, vec!["ABC", "DEF"]);
assert_eq!(total, 6);
}Iteration yields chunks from each buffer, preserving zero-copy semantics.
Chain in Network Protocols
use bytes::{Bytes, Buf, BytesMut};
// Simulating a network protocol: header + payload
struct Message {
header: Bytes,
payload: Bytes,
}
impl Message {
fn encode(&self) -> impl Buf {
// Chain header and payload without copying
self.header.clone().chain(self.payload.clone())
}
}
fn main() {
let header = Bytes::from(&[0x01, 0x02, 0x03, 0x04][..]);
let payload = Bytes::from("application data");
let message = Message { header, payload };
let encoded = message.encode();
// Process as single buffer without copying
let mut buf = encoded;
// Read header bytes
let mut header_buf = [0u8; 4];
buf.copy_to_slice(&mut header_buf);
// Read remaining as payload
let payload_len = buf.remaining();
let payload_bytes = buf.copy_to_bytes(payload_len);
println!("Header: {:?}", header_buf);
println!("Payload: {:?}", payload_bytes);
}Network protocols often have header/payload structures that benefit from zero-copy concatenation.
Chain with BytesMut
use bytes::{BytesMut, Buf};
fn main() {
let mut buf1 = BytesMut::from("Hello");
let mut buf2 = BytesMut::from("World");
// Extend without copying original data
let mut chained = buf1.split().chain(buf2.split());
// BytesMut can also be frozen to Bytes
let buf3 = BytesMut::from("!");
chained = chained.chain(buf3.freeze());
assert_eq!(chained.remaining(), 11); // "HelloWorld!"
}BytesMut integrates with chain, though freezing to Bytes is often needed.
Copy-to-bytes vs. Streaming
use bytes::{Bytes, Buf};
fn main() {
let buf1 = Bytes::from("Data ");
let buf2 = Bytes::from("Stream ");
let buf3 = Bytes::from("Example");
let chained = buf1.chain(buf2).chain(buf3);
// Option 1: Copy to contiguous bytes (allocates)
let contiguous: Bytes = chained.copy_to_bytes(chained.remaining());
// Now have one allocation with all data
// Option 2: Stream without copying (process chunk by chunk)
let mut buf1 = Bytes::from("Data ");
let buf2 = Bytes::from("Stream ");
let buf3 = Bytes::from("Example");
let mut stream = buf1.chain(buf2).chain(buf3);
while stream.has_remaining() {
let chunk = stream.chunk();
// Process chunk directly
// println!("Processing {} bytes", chunk.len());
stream.advance(chunk.len());
}
}Choose between contiguous data (with copy) or streaming (without copy).
Chain and Buf Trait Methods
use bytes::{Bytes, Buf, BufMut};
fn main() {
let buf1 = Bytes::from(&[0x12, 0x34][..]);
let buf2 = Bytes::from(&[0x56, 0x78][..]);
let chained = buf1.chain(buf2);
// Buf trait methods work across chain boundary
// Get_u16 reads 2 bytes, handling boundaries
let mut buf = chained;
let first_u16 = buf.get_u16(); // Reads 0x1234
let second_u16 = buf.get_u16(); // Reads 0x5678
// get_* methods handle cross-chunk reads
let buf1 = Bytes::from(&[0xAA][..]);
let buf2 = Bytes::from(&[0xBB, 0xCC, 0xDD][..]);
let chained = buf1.chain(buf2);
let mut buf = chained;
// get_u32 reads across boundary: 0xAABBCCDD
let value = buf.get_u32();
assert_eq!(value, 0xAABBCCDD);
}Buf trait methods transparently handle cross-chunk operations.
Performance Implications
use bytes::{Bytes, Buf};
use std::time::Instant;
fn traditional_concat(buffers: &[Bytes]) -> Bytes {
// Traditional: allocate and copy
let total_len: usize = buffers.iter().map(|b| b.len()).sum();
let mut result = Vec::with_capacity(total_len);
for buf in buffers {
result.extend_from_slice(buf);
}
Bytes::from(result)
}
fn zero_copy_chain(buffers: Vec<Bytes>) -> impl Buf {
// Zero-copy: chain without allocation
buffers.into_iter()
.fold(Bytes::new().chain(Bytes::new()), |acc, buf| acc.chain(buf))
}
fn main() {
let buffers: Vec<Bytes> = (0..100)
.map(|i| Bytes::from(format!("chunk-{} ", i)))
.collect();
// Measure traditional approach
let start = Instant::now();
let _concatenated = traditional_concat(&buffers);
let trad_duration = start.elapsed();
// Measure chain approach
let start = Instant::now();
let mut chained = zero_copy_chain(buffers);
let chain_duration = start.elapsed();
// Chain is O(1) - no allocation or copying
// Traditional is O(n) - must copy all data
println!("Traditional: {:?}", trad_duration);
println!("Chain: {:?}", chain_duration);
}chain is constant-time regardless of buffer count or size.
Chain Limitations
use bytes::{Bytes, Buf};
fn main() {
let buf1 = Bytes::from("First");
let buf2 = Bytes::from("Second");
let chained = buf1.chain(buf2);
// Limitation 1: Can only iterate forward
// No random access to bytes
// Limitation 2: chunks() doesn't exist (it's chunk() for current)
// Must manually iterate
// Limitation 3: Deep chains have overhead
// Each chain layer adds indirection
// Many chained buffers:
let deep_chain = (0..1000)
.fold(Bytes::new().chain(Bytes::new()), |acc, _| {
acc.chain(Bytes::from("x"))
});
// Reading traverses chain structure
// Consider copy_to_bytes if you'll access repeatedly
}Deep chains have performance costs; consider copying for repeated access.
Real-World Pattern: HTTP Body Handling
use bytes::{Bytes, Buf};
// Simulating hyper-style body chunks
struct Body {
chunks: Vec<Bytes>,
}
impl Body {
fn into_buf(self) -> impl Buf {
// Convert vector of chunks into a single buffer view
self.chunks.into_iter()
.fold(Bytes::new().chain(Bytes::new()), |acc, chunk| {
acc.chain(chunk)
})
}
fn into_contiguous(self) -> Bytes {
// When contiguous data is needed
let total: usize = self.chunks.iter().map(|c| c.len()).sum();
let mut buf = self.into_buf();
buf.copy_to_bytes(total)
}
}
fn main() {
let body = Body {
chunks: vec![
Bytes::from("HTTP/1.1 200 OK\r\n"),
Bytes::from("Content-Type: text/plain\r\n"),
Bytes::from("\r\n"),
Bytes::from("Hello, World!"),
],
};
// Process headers without copying
let mut buf = body.into_buf();
// Read until \r\n\r\n (simplified)
let mut headers_end = 0;
let mut found = 0;
while buf.has_remaining() {
let chunk = buf.chunk();
for (i, &byte) in chunk.iter().enumerate() {
if byte == b'\r' || byte == b'\n' {
found += 1;
if found == 4 {
// Found end of headers
break;
}
} else {
found = 0;
}
}
buf.advance(chunk.len());
}
}Network protocols benefit from chunked processing without copying.
Combining with take and other combinators
use bytes::{Bytes, Buf};
fn main() {
let buf1 = Bytes::from("12345");
let buf2 = Bytes::from("67890");
let chained = buf1.chain(buf2);
// Limit how much you read
let first_three = chained.take(3);
let mut result = vec![0u8; 3];
first_three.copy_to_slice(&mut result);
assert_eq!(&result, b"123");
// Chain with limit
let buf1 = Bytes::from("Hello");
let buf2 = Bytes::from("World");
let limited = buf1.chain(buf2).take(7);
// Read only first 7 bytes across chain boundary
let mut result = vec![0u8; 7];
limited.copy_to_slice(&mut result);
assert_eq!(&result, b"HelloWo");
}chain composes with other Buf combinators like take.
Synthesis
Quick reference:
use bytes::{Bytes, Buf};
fn main() {
// Basic chain
let buf1 = Bytes::from("Hello, ");
let buf2 = Bytes::from("World!");
let chained = buf1.chain(buf2);
// Chain multiple buffers
let buffers = vec![
Bytes::from("A"),
Bytes::from("B"),
Bytes::from("C"),
];
let multi = buffers.into_iter()
.fold(Bytes::new().chain(Bytes::new()), |acc, b| acc.chain(b));
// Operations that maintain zero-copy:
// - remaining() - total length
// - has_remaining() - check if data available
// - chunk() - current contiguous chunk
// - advance(n) - skip n bytes
// - get_*() - read integers across boundaries
// Operations that require copy:
// - copy_to_bytes(n) - allocate and copy n bytes
// - copy_to_slice(&mut [u8]) - copy to slice
// When to use chain:
// - Network data in chunks
// - Header + payload structures
// - Avoiding allocations in hot paths
// - Processing data once
// When to copy instead:
// - Need random access
// - Repeated reads of same data
// - Deep chains (many buffers)
// - Interop with APIs needing contiguous data
}Key insight: chain implements a classic "rope" data structure patternāpresenting multiple contiguous buffers as a single logical buffer without copying data. The Chain<A, B> type holds references to both underlying buffers and implements Buf by delegating to the first buffer until exhausted, then to the second. Methods like get_u32() transparently handle reading across buffer boundaries by combining bytes from both. The zero-copy property is maintained until you explicitly request contiguous data via copy_to_bytes(). This is particularly valuable in network programming where data arrives in chunks (TCP segments, HTTP body frames) that logically form a single message. Use chain when you're processing data once in sequence; use copy_to_bytes() when you need random access or will process the same data multiple times. The overhead of chain is just the Chain struct (two pointers/lengths), making it essentially free compared to allocating and copying potentially megabytes of data.