Arc<Mutex>

Guides

Rust walkthroughs

← Back to all guides

What is the purpose of quote::TokenStreamExt::append_all for extending token streams efficiently?

append_all extends a TokenStream by iterating over an iterable of TokenStreams and appending each one in sequence, providing a more ergonomic and efficient alternative to manual iteration with extend or repeated append calls when combining multiple token streams from collections. This method is essential in procedural macros where you frequently need to concatenate token streams from iterators, collections, or generated code fragments into a single output stream.

The TokenStream Extension Problem

use proc_macro2::TokenStream;
use quote::quote;
 
fn extension_problem() {
    // In procedural macros, you often need to combine multiple token streams
    // into a single output
    
    let stream1 = quote! { let x = 1; };
    let stream2 = quote! { let y = 2; };
    let stream3 = quote! { let z = 3; };
    
    // Goal: Combine these into one stream
    // Without append_all, you'd need to manually iterate and extend
    
    // Naive approach:
    let mut combined = TokenStream::new();
    combined.extend(stream1);
    combined.extend(stream2);
    combined.extend(stream3);
    
    // With append_all:
    let combined = TokenStream::new();
    let streams = vec![stream1, stream2, stream3];
    // append_all handles the iteration for you
}

Procedural macros frequently need to combine generated code fragments; append_all streamlines this pattern.

The TokenStreamExt Trait

use proc_macro2::TokenStream;
use quote::TokenStreamExt;
use quote::quote;
 
fn trait_overview() {
    // TokenStreamExt is an extension trait for proc_macro2::TokenStream
    // It provides methods for appending tokens
    
    // The trait defines:
    // - append: Append a single TokenStream
    // - append_all: Append all TokenStreams from an iterator
    // - append_separated: Append all TokenStreams with a separator
    
    let mut output = TokenStream::new();
    
    // append: Add a single token stream
    let tokens = quote! { fn foo() {} };
    output.append(tokens);
    
    // append_all: Add multiple token streams from an iterable
    let functions = vec![
        quote! { fn a() {} },
        quote! { fn b() {} },
        quote! { fn c() {} },
    ];
    output.append_all(functions);
    
    // The result combines all streams in order
    println!("{}", output);
}

TokenStreamExt provides three methods: append, append_all, and append_separated.

Basic append_all Usage

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn basic_usage() {
    let mut output = TokenStream::new();
    
    // Start with some initial tokens
    output.extend(quote! {
        mod my_module {
    });
    
    // Append multiple function definitions
    let functions = vec![
        quote! { fn add(a: i32, b: i32) -> i32 { a + b } },
        quote! { fn sub(a: i32, b: i32) -> i32 { a - b } },
        quote! { fn mul(a: i32, b: i32) -> i32 { a * b } },
    ];
    
    output.append_all(functions);
    
    // Close the module
    output.extend(quote! { });
    
    println!("{}", output);
    // Output:
    // mod my_module {
    //     fn add(a: i32, b: i32) -> i32 { a + b }
    //     fn sub(a: i32, b: i32) -> i32 { a - b }
    //     fn mul(a: i32, b: i32) -> i32 { a * b }
    // }
}

append_all takes any iterable and appends each element in order.

append_all vs Manual Iteration

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn comparison() {
    let functions: Vec<TokenStream> = (0..3)
        .map(|i| quote! { fn func#i() {} })
        .collect();
    
    // Manual iteration with extend
    let mut output1 = TokenStream::new();
    for func in &functions {
        output1.extend(func.clone());
    }
    
    // Manual iteration with append
    let mut output2 = TokenStream::new();
    for func in functions.clone() {
        output2.append(func);
    }
    
    // Using append_all (most ergonomic)
    let mut output3 = TokenStream::new();
    output3.append_all(&functions);
    
    // All three produce equivalent results
    // append_all is clearer and more idiomatic
    
    // Performance: append_all may be more efficient because:
    // 1. It can pre-calculate the total size in some cases
    // 2. It's implemented with a single call into proc_macro2
    // 3. The intent is clear to the compiler for optimization
}

append_all is more ergonomic than manual iteration and may have performance benefits.

Working with Iterators

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn with_iterators() {
    // append_all works with any iterable, not just Vec
    
    // Generate field getters from struct fields
    let fields = ["name", "age", "email"];
    
    let mut getters = TokenStream::new();
    getters.append_all(fields.iter().map(|field| {
        let field_name = syn::Ident::new(field, proc_macro2::Span::call_site());
        quote! {
            pub fn #field_name(&self) -> &str {
                &self.#field_name
            }
        }
    }));
    
    // Works with iterators directly
    let mut output = TokenStream::new();
    output.append_all((0..5).map(|i| {
        quote! { const VALUE_#i: i32 = #i; }
    }));
    
    // Works with Option (iterates 0 or 1 times)
    let optional_stream: Option<TokenStream> = Some(quote! { fn optional() {} });
    output.append_all(optional_stream);
}

append_all accepts any IntoIterator, including iterators, slices, and options.

append_all in Derive Macros

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
use syn::{parse_macro_input, DeriveInput, Data, Fields};
 
// A derive macro that generates getter methods for struct fields
fn derive_getters_example(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    let input = parse_macro_input!(input as DeriveInput);
    
    let struct_name = &input.ident;
    
    // Extract field information
    let fields = match &input.data {
        Data::Struct(data) => match &data.fields {
            Fields::Named(fields) => &fields.named,
            _ => panic!("Only named fields supported"),
        },
        _ => panic!("Only structs supported"),
    };
    
    // Generate getter for each field
    let getters: Vec<TokenStream> = fields
        .iter()
        .map(|field| {
            let field_name = field.ident.as_ref().unwrap();
            let field_type = &field.ty;
            quote! {
                pub fn #field_name(&self) -> &#field_type {
                    &self.#field_name
                }
            }
        })
        .collect();
    
    // Use append_all to combine all getters
    let mut output = TokenStream::new();
    output.extend(quote! {
        impl #struct_name {
    });
    output.append_all(getters);
    output.extend(quote! { } });
    
    output.into()
}

Derive macros commonly generate code for each struct field; append_all collects the results.

append_all with Collections

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn with_collections() {
    // Vec<TokenStream>
    let items = vec![
        quote! { let a = 1; },
        quote! { let b = 2; },
        quote! { let c = 3; },
    ];
    
    let mut output = TokenStream::new();
    output.append_all(&items);  // Works with references
    output.append_all(items);   // Works with owned values
    
    // Arrays
    let array = [
        quote! { fn x() {} },
        quote! { fn y() {} },
        quote! { fn z() {} },
    ];
    output.append_all(array);
    
    // Slices
    let slice: &[TokenStream] = &array;
    output.append_all(slice);
    
    // HashMap (iterates in arbitrary order)
    use std::collections::HashMap;
    let mut map = HashMap::new();
    map.insert("one", quote! { const ONE: i32 = 1; });
    map.insert("two", quote! { const TWO: i32 = 2; });
    output.append_all(map.values());
}

append_all works with any collection that implements IntoIterator.

append_separated for Delimited Items

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn separated_items() {
    // Related: append_separated adds a separator between items
    // This is useful for comma-separated lists, etc.
    
    let items = vec![
        quote! { "apple" },
        quote! { "banana" },
        quote! { "cherry" },
    ];
    
    // Create an array with comma separators
    let mut output = TokenStream::new();
    output.extend(quote! { let fruits = [ });
    output.append_separated(items, quote! { , });
    output.extend(quote! { ]; });
    
    println!("{}", output);
    // Output: let fruits = [ "apple" , "banana" , "cherry" ];
    
    // Compare to append_all (no separator):
    let items = vec![
        quote! { "a" },
        quote! { "b" },
        quote! { "c" },
    ];
    let mut output2 = TokenStream::new();
    output2.append_all(items);
    // Output: "a" "b" "c" (no separator between)
}

append_separated is a sibling method that inserts separators; append_all just concatenates.

Generating Match Arms

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn match_arms_example() {
    // Generate match arms from a list of variants
    let variants = ["Red", "Green", "Blue"];
    
    // Generate match arms with append_all
    let mut arms = TokenStream::new();
    arms.append_all(variants.iter().map(|v| {
        let variant = syn::Ident::new(v, proc_macro2::Span::call_site());
        quote! { Color::#variant => stringify!(#variant), }
    }));
    
    let output = quote! {
        fn color_name(color: Color) -> &'static str {
            match color {
                #arms
            }
        }
    };
    
    // Result:
    // fn color_name(color: Color) -> &'static str {
    //     match color {
    //         Color::Red => stringify!(Red),
    //         Color::Green => stringify!(Green),
    //         Color::Blue => stringify!(Blue),
    //     }
    // }
}

Generating match arms from data structures is a common append_all use case.

Generating Struct Fields

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn struct_fields_example() {
    // Generate struct fields from configuration
    let fields = [
        ("id", "u64"),
        ("name", "String"),
        ("active", "bool"),
    ];
    
    let mut field_tokens = TokenStream::new();
    field_tokens.append_all(fields.iter().map(|(name, ty)| {
        let field_name = syn::Ident::new(name, proc_macro2::Span::call_site());
        let field_type = syn::Ident::new(ty, proc_macro2::Span::call_site());
        quote! { #field_name: #field_type, }
    }));
    
    let output = quote! {
        struct Record {
            #field_tokens
        }
    };
    
    // Result:
    // struct Record {
    //     id: u64,
    //     name: String,
    //     active: bool,
    // }
}

Dynamic struct generation benefits from append_all to combine field definitions.

Nesting and Complex Generation

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn nested_generation() {
    // Generate nested structures
    let modules = ["network", "storage", "auth"];
    
    let mut output = TokenStream::new();
    
    output.append_all(modules.iter().map(|module_name| {
        let module = syn::Ident::new(module_name, proc_macro2::Span::call_site());
        
        // Generate functions for each module
        let functions: Vec<TokenStream> = (0..3)
            .map(|i| {
                let fn_name = syn::Ident::new(
                    &format!("func_{}", i),
                    proc_macro2::Span::call_site()
                );
                quote! { pub fn #fn_name() {} }
            })
            .collect();
        
        // Combine function streams using append_all
        let mut func_stream = TokenStream::new();
        func_stream.append_all(functions);
        
        quote! {
            pub mod #module {
                #func_stream
            }
        }
    }));
    
    // Result: Three modules, each with three functions
}

Nested generation uses append_all at each level to combine generated code.

append_all with Filtered Iterators

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn filtered_generation() {
    let fields = [
        ("id", Some("u64")),
        ("name", Some("String")),
        ("internal", None),  // Skip this field
        ("active", Some("bool")),
    ];
    
    let mut output = TokenStream::new();
    
    // append_all works with filtered iterators
    output.append_all(
        fields
            .iter()
            .filter(|(_, ty)| ty.is_some())
            .map(|(name, ty)| {
                let field_name = syn::Ident::new(name, proc_macro2::Span::call_site());
                let field_type = ty.unwrap();
                let type_ident = syn::Ident::new(field_type, proc_macro2::Span::call_site());
                quote! { #field_name: #type_ident, }
            })
    );
    
    // Only id, name, and active fields generated
}

append_all works with any iterator transformation, including filtering.

Comparing append vs append_all vs extend

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn method_comparison() {
    let mut stream = TokenStream::new();
    let a = quote! { let a = 1; };
    let b = quote! { let b = 2; };
    let items = vec![quote! { let c = 3; }, quote! { let d = 4; }];
    
    // append: Add a single TokenStream
    // (from TokenStreamExt trait)
    stream.append(a);
    stream.append(b);
    
    // extend: Add tokens from a TokenStream (standard library method)
    // This consumes the stream being extended
    let more = quote! { let e = 5; };
    stream.extend(more);
    
    // append_all: Add multiple TokenStreams from an iterable
    // (from TokenStreamExt trait)
    stream.append_all(&items);
    
    // Key differences:
    // - append: Single TokenStream, clear intent
    // - extend: Standard trait method, works with IntoIterator<TokenTree>
    // - append_all: Multiple TokenStreams, ergonomic for collections
    
    // append_all vs extend:
    // - extend works on TokenTree iterators
    // - append_all works on TokenStream iterators
    // - append_all is more convenient for TokenStream collections
    
    // Performance note:
    // append_all internally calls extend on each stream
    // It's ergonomic sugar, not a fundamentally different operation
}

append_all is specialized for collections of TokenStream; extend works on token tree iterators.

Memory and Performance Considerations

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn performance_notes() {
    // TokenStream is designed for efficient concatenation
    // append_all is implemented to minimize allocations
    
    // When you do:
    let items: Vec<TokenStream> = (0..100)
        .map(|i| quote! { let _#i = #i; })
        .collect();
    
    let mut output = TokenStream::new();
    output.append_all(&items);
    
    // This is more efficient than:
    // let output = items.into_iter().fold(TokenStream::new(), |mut acc, s| {
    //     acc.extend(s);
    //     acc
    // });
    
    // Because append_all can operate on the underlying token structure
    // without intermediate conversions
    
    // Memory: TokenStream internally stores a Vec<TokenTree>
    // Appending extends this vector
    // Multiple appends may cause reallocations
    // But proc_macro2::TokenStream is optimized for this pattern
    
    // For very large code generation, consider:
    // - Building in chunks
    // - Using quote!{} directly for complex structures
    // - Avoiding unnecessary cloning
}

TokenStream is optimized for extension; append_all provides idiomatic collection handling.

Working with Generics

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn generics_example() {
    // Generate implementations for multiple types
    let types = ["i32", "i64", "u32", "u64"];
    
    let mut impls = TokenStream::new();
    impls.append_all(types.iter().map(|ty| {
        let type_ident = syn::Ident::new(ty, proc_macro2::Span::call_site());
        quote! {
            impl MyTrait for #type_ident {
                fn process(&self) -> bool {
                    true
                }
            }
        }
    }));
    
    // Result: Four impl blocks
    
    // Generate where clauses for generic implementations
    let constraints = ["Debug", "Clone", "Serialize"];
    let mut where_clause = TokenStream::new();
    where_clause.append_all(constraints.iter().map(|c| {
        let trait_name = syn::Ident::new(c, proc_macro2::Span::call_site());
        quote! { T: #trait_name, }
    }));
    
    let impl_block = quote! {
        impl<T> MyTrait for T
        where
            #where_clause
        {
            fn process(&self) -> bool {
                true
            }
        }
    };
}

Generic code generation uses append_all to combine type bounds and implementations.

Error Handling in Code Generation

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
fn error_handling() {
    // When generating code that might fail validation
    let inputs = ["valid_name", "123_invalid", "another_valid", "also!bad"];
    
    // Filter and validate, then generate
    let valid_tokens: Vec<TokenStream> = inputs
        .iter()
        .filter(|name| name.chars().next().unwrap().is_alphabetic())
        .filter(|name| name.chars().all(|c| c.is_alphanumeric() || c == '_'))
        .map(|name| {
            let ident = syn::Ident::new(name, proc_macro2::Span::call_site());
            quote! { let #ident = 0; }
        })
        .collect();
    
    // append_all handles empty iterators gracefully
    let mut output = TokenStream::new();
    output.append_all(valid_tokens);  // Works even if empty
    
    // Option<TokenStream> can be appended (iterates 0 or 1 times)
    let optional: Option<TokenStream> = None;
    output.append_all(optional);  // No-op
    
    let optional: Option<TokenStream> = Some(quote! { let x = 1; });
    output.append_all(optional);  // Appends the stream
}

append_all handles empty collections and options gracefully.

Complete Example: Builder Derive

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
use syn::{parse_macro_input, DeriveInput, Data, Fields, Ident};
 
fn derive_builder_example(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    let input = parse_macro_input!(input as DeriveInput);
    
    let struct_name = &input.ident;
    let builder_name = Ident::new(
        &format!("{}Builder", struct_name),
        struct_name.span(),
    );
    
    // Extract fields
    let fields = match &input.data {
        Data::Struct(data) => match &data.fields {
            Fields::Named(fields) => &fields.named,
            _ => panic!("Only named fields supported"),
        },
        _ => panic!("Only structs supported"),
    };
    
    // Generate builder fields (Option<T> for each)
    let builder_fields: Vec<TokenStream> = fields
        .iter()
        .map(|field| {
            let field_name = field.ident.as_ref().unwrap();
            let field_type = &field.ty;
            quote! { #field_name: std::option::Option<#field_type> }
        })
        .collect();
    
    // Generate setter methods
    let setters: Vec<TokenStream> = fields
        .iter()
        .map(|field| {
            let field_name = field.ident.as_ref().unwrap();
            let field_type = &field.ty;
            quote! {
                pub fn #field_name(mut self, value: #field_type) -> Self {
                    self.#field_name = Some(value);
                    self
                }
            }
        })
        .collect();
    
    // Generate build method field assignments
    let build_fields: Vec<TokenStream> = fields
        .iter()
        .map(|field| {
            let field_name = field.ident.as_ref().unwrap();
            quote! {
                #field_name: self.#field_name.take().ok_or(concat!("missing field: ", stringify!(#field_name)))?
            }
        })
        .collect();
    
    // Combine everything using append_all
    let mut all_setters = TokenStream::new();
    all_setters.append_all(setters);
    
    let mut all_build_fields = TokenStream::new();
    all_build_fields.append_separated(build_fields, quote! { , });
    
    let output = quote! {
        pub struct #builder_name {
            #(#builder_fields),*
        }
        
        impl #builder_name {
            #all_setters
            
            pub fn build(self) -> Result<#struct_name, &'static str> {
                Ok(#struct_name {
                    #all_build_fields
                })
            }
        }
        
        impl #struct_name {
            pub fn builder() -> #builder_name {
                #builder_name {
                    #(#fields.ident: None),*
                }
            }
        }
    };
    
    output.into()
}

A complete derive macro uses append_all to combine generated methods and field assignments.

Summary Table

fn summary() {
    // | Method            | Purpose                          | Input Type           |
    // |-------------------|----------------------------------|----------------------|
    // | append            | Add single TokenStream           | TokenStream          |
    // | append_all        | Add multiple TokenStreams        | IntoIterator<Item=T> |
    // | append_separated  | Add multiple with separator      | IntoIterator + Token |
    // | extend            | Add TokenTrees from iterator     | IntoIterator<Token>  |
    
    // | Use Case                          | Best Method      |
    // |-----------------------------------|------------------|
    // | Single stream                     | append           |
    // | Vec<TokenStream>                  | append_all       |
    // | Iterator generating streams       | append_all       |
    // | Comma-separated list              | append_separated |
    // | TokenTree iterator                | extend           |
    
    // | Performance Consideration                         | Impact            |
    // |---------------------------------------------------|-------------------|
    // | append_all vs manual iteration                   | Minimal diff      |
    // | Very large collections                            | Consider chunking |
    // | Cloning TokenStreams in append_all                | Usually necessary |
}

Synthesis

Quick reference:

use proc_macro2::TokenStream;
use quote::{quote, TokenStreamExt};
 
let mut output = TokenStream::new();
 
// Single stream
output.append(quote! { fn single() {} });
 
// Multiple streams from collection
let items = vec![
    quote! { fn a() {} },
    quote! { fn b() {} },
    quote! { fn c() {} },
];
output.append_all(&items);
 
// From iterator
output.append_all((0..3).map(|i| quote! { const NUM_#i: i32 = #i; }));
 
// With separator
output.append_separated(
    [quote! { 1 }, quote! { 2 }, quote! { 3 }],
    quote! { , }
);

Key insight: append_all is an ergonomic extension method that simplifies combining multiple TokenStreams into one. Without it, you'd write manual iteration:

// Without append_all:
let mut output = TokenStream::new();
for stream in streams {
    output.extend(stream);
}
 
// With append_all:
output.append_all(streams);

The method works with any IntoIterator, making it convenient for vectors, slices, arrays, and iterators—particularly those returned by map() transformations in derive macros. It's part of the quote ecosystem's design for procedural macros, where code generation frequently produces multiple token streams that need combining. The related append_separated method adds delimiters between items (useful for comma-separated function arguments, struct fields, or match arms). Use append for single streams, append_all for collections and iterators, and append_separated when you need delimiters between items. The underlying operation is efficient because proc_macro2::TokenStream is designed for extension, but for extremely large code generation, consider building in logical chunks rather than one massive append_all chain.