Arc<Mutex>

Guides

Rust walkthroughs

← Back to all guides

How does quote::ToTokens::to_tokens enable code generation from custom types in procedural macros?

ToTokens::to_tokens is the foundational mechanism by which Rust values become TokenStream fragments for code generation. The trait provides a uniform interface: any type implementing ToTokens can append its token representation to an existing TokenStream, enabling compositional code generation. The quote! macro leverages this trait—when you interpolate #value inside quote!, the macro calls value.to_tokens(). By implementing ToTokens for custom types, your types become first-class citizens in macro output, allowing them to participate in code generation just like built-in types. This creates a powerful abstraction: procedural macros can transform custom syntax elements directly into valid Rust code without manually building token streams.

The ToTokens Trait Definition

use quote::quote;
use proc_macro2::TokenStream;
 
// The ToTokens trait is defined as:
pub trait ToTokens {
    fn to_tokens(&self, tokens: &mut TokenStream);
    
    // Provided methods:
    fn to_token_stream(&self) -> TokenStream { ... }
    fn into_token_stream(self) -> TokenStream where Self: Sized { ... }
}
 
// The key insight:
// - to_tokens APPENDS to an existing TokenStream
// - This enables chaining and composition
// - The TokenStream is the target, not the return value
 
fn main() {
    let name = "example";
    let tokens = quote! {
        fn #name() {}
    };
    println!("{}", tokens);
}

The trait's signature—appending to a mutable TokenStream rather than returning one—enables efficient composition.

How quote! Uses ToTokens

use quote::quote;
use proc_macro2::TokenStream;
 
fn main() {
    let value = 42u32;
    
    // When you write #value in quote!, the macro expands to:
    let tokens = quote! {
        let x = #value;
    };
    
    // Conceptually, this becomes:
    let mut stream = TokenStream::new();
    stream.extend(quote!(let x = ));
    value.to_tokens(&mut stream);  // ToTokens::to_tokens called here
    stream.extend(quote!(;));
    
    // Any type implementing ToTokens can be interpolated:
    let ident = quote::format_ident!("my_var");
    let ty: syn::Type = syn::parse_quote!(i32);
    let lit = proc_macro2::Literal::i32_unsuffixed(42);
    
    let tokens = quote! {
        let #ident: #ty = #lit;
    };
    
    println!("{}", tokens);
}

The quote! macro calls to_tokens for every #interpolation, delegating the actual token generation to each type.

Built-in ToTokens Implementations

use quote::quote;
use proc_macro2::{TokenStream, Ident, Literal, Span};
 
fn main() {
    // Many types already implement ToTokens:
    
    // Primitives
    let int: i32 = 42;
    let float: f64 = 3.14;
    let bool_val: bool = true;
    let str_val: &str = "hello";
    
    let tokens = quote! {
        #int #float #bool_val #str_val
    };
    // Generates: 42 3.14 true "hello"
    
    // proc_macro2 types
    let ident = Ident::new("my_function", Span::call_site());
    let literal = Literal::string("world");
    
    let tokens = quote! {
        fn #ident() -> &'static str {
            #literal
        }
    };
    // Generates: fn my_function() -> &'static str { "world" }
    
    // syn types (from parsed Rust code)
    // let ty: syn::Type = syn::parse_quote!(Option<String>);
    // let path: syn::Path = syn::parse_quote!(std::collections::HashMap);
    
    // Collections
    let items = vec
![quote!(item1), quote!(item2)];
    let tokens = quote! {
        #(#items),*
    };
    // Generates: item1 , item2
}

quote provides implementations for primitives, proc_macro2 types, syn types, and collections.

Implementing ToTokens for Custom Types

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
// A custom type representing a function signature
struct FunctionDef {
    name: String,
    params: Vec<(String, String)>,  // (name, type)
    return_type: String,
}
 
impl ToTokens for FunctionDef {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        // Generate: fn <name>(<params>) -> <return_type> { ... }
        
        tokens.extend(quote!(fn));
        tokens.extend(quote!(#self.name));
        tokens.extend(quote!());
        
        // Add parameters
        tokens.extend(quote!());
        for (i, (param_name, param_type)) in self.params.iter().enumerate() {
            if i > 0 {
                tokens.extend(quote!(,));
            }
            tokens.extend(quote!(#param_name: #param_type));
        }
        tokens.extend(quote!());
        
        // Add return type
        tokens.extend(quote!(-> #self.return_type));
    }
}
 
fn main() {
    let func = FunctionDef {
        name: "process".to_string(),
        params: vec
![("input".to_string(), "String".to_string())],
        return_type: "i32".to_string(),
    };
    
    let tokens = quote! {
        #func
    };
    
    println!("{}", tokens);
}

Implementing ToTokens allows your types to be interpolated directly in quote! macros.

A Practical Example: Struct Generator

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
 
// Representing a field in a generated struct
struct Field {
    name: String,
    ty: String,
}
 
impl ToTokens for Field {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        tokens.extend(quote! {
            #name: #ty,
        });
    }
}
 
// Representing the entire struct
struct StructDef {
    name: String,
    fields: Vec<Field>,
}
 
impl ToTokens for StructDef {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let fields = &self.fields;
        
        tokens.extend(quote! {
            struct #name {
                #(#fields)*
            }
        });
    }
}
 
fn main() {
    let struct_def = StructDef {
        name: "Person".to_string(),
        fields: vec
![
            Field { name: "name".to_string(), ty: "String".to_string() },
            Field { name: "age".to_string(), ty: "u32".to_string() },
        ],
    };
    
    let tokens = quote! {
        #struct_def
    };
    
    println!("{}", tokens);
    // Output:
    // struct Person {
    //     name: String,
    //     age: u32,
    // }
}

Complex code generation becomes compositional when each type knows how to convert itself to tokens.

ToTokens with syn Types

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::{Ident, Type, Path, parse_quote};
 
// syn types already implement ToTokens
// This enables seamless integration with parsed Rust code
 
struct TypedVariable {
    name: Ident,
    ty: Type,
    value: proc_macro2::Literal,
}
 
impl ToTokens for TypedVariable {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        let value = &self.value;
        
        tokens.extend(quote! {
            let #name: #ty = #value;
        });
    }
}
 
fn main() {
    // Using syn types that implement ToTokens
    let var = TypedVariable {
        name: parse_quote!(count),
        ty: parse_quote!(i32),
        value: proc_macro2::Literal::i32_unsuffixed(42),
    };
    
    // The quote! macro automatically uses each type's ToTokens implementation
    let tokens = quote! {
        #var
        println!("{}")
, count);
    };
    
    println!("{}", tokens);
}

syn types implement ToTokens, allowing parsed code to be regenerated or transformed.

TokenStreamAppend Trait Extension

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
// ToTokens works with TokenStreamExt for convenient building
 
fn main() {
    let mut tokens = TokenStream::new();
    
    // TokenStreamExt provides append_all for iterators
    let items = vec
![quote!(item1), quote!(item2), quote!(item3)];
    tokens.append_all(items);
    
    // Equivalent to:
    let mut tokens2 = TokenStream::new();
    for item in items {
        item.to_tokens(&mut tokens2);
    }
    
    // TokenStreamExt::append also exists for single items
    let mut tokens3 = TokenStream::new();
    tokens3.append(quote!(struct Foo;));
    
    println!("{}", tokens);
    println!("{}", tokens2);
    println!("{}", tokens3);
}

TokenStreamExt provides helper methods that internally use ToTokens.

Implementing ToTokens for Enums

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::Ident;
 
enum ValueType {
    Integer(i32),
    Float(f64),
    String(String),
    Boolean(bool),
}
 
impl ToTokens for ValueType {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        match self {
            ValueType::Integer(n) => {
                let lit = proc_macro2::Literal::i32_unsuffixed(*n);
                tokens.extend(quote!(#lit));
            }
            ValueType::Float(f) => {
                let lit = proc_macro2::Literal::f64_unsuffixed(*f);
                tokens.extend(quote!(#lit));
            }
            ValueType::String(s) => {
                let lit = proc_macro2::Literal::string(s);
                tokens.extend(quote!(#lit));
            }
            ValueType::Boolean(b) => {
                tokens.extend(quote!(#b));
            }
        }
    }
}
 
fn main() {
    let values = vec
![
        ValueType::Integer(42),
        ValueType::Float(3.14),
        ValueType::String("hello".to_string()),
        ValueType::Boolean(true),
    ];
    
    let tokens = quote! {
        let values = [#(#values),*];
    };
    
    println!("{}", tokens);
    // Output: let values = [42, 3.14, "hello", true];
}

Enums can dispatch to appropriate token representations based on variant.

Repetition Patterns with ToTokens

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::{Ident, parse_quote};
 
struct FunctionParams {
    params: Vec<(Ident, syn::Type)>,
}
 
impl ToTokens for FunctionParams {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        // Generate comma-separated parameters
        let params = &self.params;
        tokens.extend(quote! {
            #(#params.0: #params.1),*
        });
        // This doesn't work directly - need to iterate properly
    }
}
 
// Better approach using TokenStreamExt
impl ToTokens for FunctionParams {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        use quote::TokenStreamExt;
        
        tokens.append_all(self.params.iter().enumerate().map(|(i, (name, ty))| {
            if i > 0 {
                quote!(, #name: #ty)
            } else {
                quote!(#name: #ty)
            }
        }));
    }
}
 
fn main() {
    let params = FunctionParams {
        params: vec
![
            (parse_quote!(a), parse_quote!(i32)),
            (parse_quote!(b), parse_quote!(String)),
            (parse_quote!(c), parse_quote!(bool)),
        ],
    };
    
    let tokens = quote! {
        fn example(#params) {}
    };
    
    println!("{}", tokens);
}

Complex repetition patterns require careful implementation of to_tokens.

Conditional Token Generation

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
 
struct OptionalField {
    name: String,
    ty: String,
    is_public: bool,
    has_default: bool,
}
 
impl ToTokens for OptionalField {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        
        // Conditionally include visibility
        if self.is_public {
            tokens.extend(quote!(pub));
        }
        
        tokens.extend(quote!(#name: #ty));
        
        // Conditionally include default attribute
        if self.has_default {
            tokens.extend(quote!(#[default]));
        }
    }
}
 
fn main() {
    let fields = vec
![
        OptionalField {
            name: "id".to_string(),
            ty: "u32".to_string(),
            is_public: true,
            has_default: false,
        },
        OptionalField {
            name: "name".to_string(),
            ty: "String".to_string(),
            is_public: true,
            has_default: true,
        },
        OptionalField {
            name: "internal".to_string(),
            ty: "bool".to_string(),
            is_public: false,
            has_default: false,
        },
    ];
    
    let tokens = quote! {
        struct Data {
            #(#fields),*
        }
    };
    
    println!("{}", tokens);
}

ToTokens implementations can contain arbitrary conditional logic for token generation.

Integration in Procedural Macros

// In a procedural macro crate:
use proc_macro::TokenStream;
use quote::{quote, ToTokens};
use syn::{parse_macro_input, DeriveInput, Ident};
 
// This would be in a procedural macro crate
// The following shows the pattern:
 
struct MyStruct {
    name: Ident,
    fields: Vec<Ident>,
}
 
impl ToTokens for MyStruct {
    fn to_tokens(&self, tokens: &mut proc_macro2::TokenStream) {
        let name = &self.name;
        let fields = &self.fields;
        
        tokens.extend(quote! {
            struct #name {
                #(#fields: String),*
            }
        });
    }
}
 
// Example derive macro pattern:
// fn my_derive_macro(input: TokenStream) -> TokenStream {
//     let input = parse_macro_input!(input as DeriveInput);
//     let name = &input.ident;
//     
//     // Create custom representation
//     let my_struct = MyStruct { /* ... */ };
//     
//     // Generate output using ToTokens
//     let output = quote! {
//         #my_struct
//         
//         impl #name {
//             fn new() -> Self { /* ... */ }
//         }
//     };
//     
//     output.into()
// }
 
fn main() {
    // Demonstration of the pattern
    let my_struct = MyStruct {
        name: syn::parse_quote!(Person),
        fields: vec
![syn::parse_quote!(name), syn::parse_quote!(age)],
    };
    
    let tokens = quote! {
        #my_struct
    };
    
    println!("{}", tokens);
}

ToTokens is the bridge between your macro's internal representation and generated code.

ToTokens vs Display vs Debug

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use std::fmt;
 
struct NamedValue {
    name: String,
    value: i32,
}
 
// ToTokens: For code generation (produces valid Rust tokens)
impl ToTokens for NamedValue {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let value = self.value;
        tokens.extend(quote! {
            #name: #value
        });
    }
}
 
// Display: For user-facing output (produces human-readable text)
impl fmt::Display for NamedValue {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{} = {}", self.name, self.value)
    }
}
 
// Debug: For debugging (produces developer-readable output)
impl fmt::Debug for NamedValue {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.debug_struct("NamedValue")
            .field("name", &self.name)
            .field("value", &self.value)
            .finish()
    }
}
 
fn main() {
    let nv = NamedValue {
        name: "count".to_string(),
        value: 42,
    };
    
    // ToTokens: Generates valid Rust code
    let tokens = quote! { #nv };
    println!("ToTokens: {}", tokens);  // count: 42
    
    // Display: Human-readable
    println!("Display: {}", nv);  // count = 42
    
    // Debug: Structured representation
    println!("Debug: {:?}", nv);  // NamedValue { name: "count", value: 42 }
}

ToTokens is specifically for code generation; Display and Debug serve different purposes.

Nested Token Generation

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::{Ident, parse_quote};
 
struct Block {
    statements: Vec<Statement>,
}
 
impl ToTokens for Block {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        tokens.extend(quote! {{});
        for stmt in &self.statements {
            stmt.to_tokens(tokens);
        }
        tokens.extend(quote! {}});
    }
}
 
struct Statement {
    expr: Expr,
}
 
impl ToTokens for Statement {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        self.expr.to_tokens(tokens);
        tokens.extend(quote!(;));
    }
}
 
enum Expr {
    Literal(i32),
    Variable(Ident),
    Call { func: Ident, args: Vec<Expr> },
}
 
impl ToTokens for Expr {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        match self {
            Expr::Literal(n) => {
                let lit = proc_macro2::Literal::i32_suffixed(*n);
                tokens.extend(quote!(#lit));
            }
            Expr::Variable(name) => {
                tokens.extend(quote!(#name));
            }
            Expr::Call { func, args } => {
                tokens.extend(quote!(#func(#(#args),*)));
            }
        }
    }
}
 
fn main() {
    let block = Block {
        statements: vec
![
            Statement {
                expr: Expr::Call {
                    func: parse_quote!(process),
                    args: vec
![Expr::Literal(42), Expr::Variable(parse_quote!(x))],
                },
            },
            Statement {
                expr: Expr::Variable(parse_quote!(result)),
            },
        ],
    };
    
    let tokens = quote! {
        fn wrapper() #block
    };
    
    println!("{}", tokens);
}

Compositional token generation allows building complex ASTs that generate correct code.

Using Tokens for AST Transformation

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::{ItemFn, parse_quote};
 
// Transform function items by adding attributes
struct InstrumentedFn {
    inner: ItemFn,
}
 
impl ToTokens for InstrumentedFn {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let inner = &self.inner;
        let name = &inner.sig.ident;
        
        // Add instrumentation wrapper
        tokens.extend(quote! {
            #[allow(unused)]
            #inner
            
            fn #name() {
                println!("Entering function");
                // ... original implementation
            }
        });
    }
}
 
fn main() {
    let original_fn: ItemFn = parse_quote! {
        fn calculate(x: i32) -> i32 {
            x * 2
        }
    };
    
    let instrumented = InstrumentedFn {
        inner: original_fn,
    };
    
    let tokens = quote! {
        #instrumented
    };
    
    println!("{}", tokens);
}

ToTokens enables AST transformation by regenerating modified structures.

TokenStreamExt for Ergonomic Building

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
fn main() {
    let mut tokens = TokenStream::new();
    
    // Using TokenStreamExt methods:
    
    // append: Add a single item implementing ToTokens
    tokens.append(quote!(struct));
    tokens.append(quote::format_ident!("MyStruct"));
    tokens.append(quote!({}));
    
    println!("After append: {}", tokens);
    
    // append_all: Add all items from an iterator
    let fields = vec
!["a", "b", "c"];
    tokens.clear();
    tokens.append_all(fields.iter().map(|f| {
        let f = syn::Ident::new(f, proc_macro2::Span::call_site());
        quote!(#f: i32)
    }));
    
    println!("After append_all: {}", tokens);
    
    // append_separated: Add items with separator
    tokens.clear();
    tokens.append_separated(fields.iter(), quote!(,));
    
    println!("After append_separated: {}", tokens);
}

TokenStreamExt provides ergonomic methods that internally use ToTokens.

Complete Example: Code Generator

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::{Ident, Type, parse_quote};
 
// A complete code generator using ToTokens
 
struct GeneratedModule {
    name: Ident,
    structs: Vec<GeneratedStruct>,
    functions: Vec<GeneratedFunction>,
}
 
impl ToTokens for GeneratedModule {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let structs = &self.structs;
        let functions = &self.functions;
        
        tokens.extend(quote! {
            mod #name {
                #(#structs)*
                #(#functions)*
            }
        });
    }
}
 
struct GeneratedStruct {
    name: Ident,
    fields: Vec<GeneratedField>,
}
 
impl ToTokens for GeneratedStruct {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let fields = &self.fields;
        
        tokens.extend(quote! {
            struct #name {
                #(#fields),*
            }
        });
    }
}
 
struct GeneratedField {
    name: Ident,
    ty: Type,
}
 
impl ToTokens for GeneratedField {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        tokens.extend(quote!(#name: #ty));
    }
}
 
struct GeneratedFunction {
    name: Ident,
    params: Vec<GeneratedParam>,
    return_type: Type,
    body: TokenStream,
}
 
impl ToTokens for GeneratedFunction {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let params = &self.params;
        let return_type = &self.return_type;
        let body = &self.body;
        
        tokens.extend(quote! {
            fn #name(#(#params),*) -> #return_type #body
        });
    }
}
 
struct GeneratedParam {
    name: Ident,
    ty: Type,
}
 
impl ToTokens for GeneratedParam {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        tokens.extend(quote!(#name: #ty));
    }
}
 
fn main() {
    let module = GeneratedModule {
        name: parse_quote!(data),
        structs: vec
![
            GeneratedStruct {
                name: parse_quote!(Person),
                fields: vec
![
                    GeneratedField { name: parse_quote!(name), ty: parse_quote!(String) },
                    GeneratedField { name: parse_quote!(age), ty: parse_quote!(u32) },
                ],
            },
        ],
        functions: vec
![
            GeneratedFunction {
                name: parse_quote!(create_person),
                params: vec
![
                    GeneratedParam { name: parse_quote!(name), ty: parse_quote!(String) },
                    GeneratedParam { name: parse_quote!(age), ty: parse_quote!(u32) },
                ],
                return_type: parse_quote!(Person),
                body: quote!({ Person { name, age } }),
            },
        ],
    };
    
    let tokens = quote! {
        #module
    };
    
    println!("{}", tokens);
}

A complete code generation system uses ToTokens at every level for compositional assembly.

Synthesis

Core mechanism:

  • ToTokens::to_tokens appends tokens to an existing TokenStream
  • quote! macro calls to_tokens for every #interpolation
  • Types implementing ToTokens become first-class code generation targets

Key implementation pattern:

impl ToTokens for MyType {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        // Extend tokens with this type's representation
        tokens.extend(quote! { /* my tokens */ });
    }
}

When to implement ToTokens:

  • Custom types that need to appear in generated code
  • AST transformations that regenerate modified structures
  • Types representing configuration that drives code generation
  • Wrapper types that add context or attributes to existing types

Compositional power:

  • Each type handles its own token generation
  • Complex structures delegate to simpler ones
  • quote! handles repetition with #(#items),* patterns
  • TokenStreamExt provides ergonomic building methods

Key insight: ToTokens::to_tokens is the fundamental building block for procedural macro code generation. By implementing this trait for custom types, you create compositional, maintainable code generators where each type knows how to express itself as valid Rust tokens. The quote! macro serves as a convenient interface that internally calls to_tokens for interpolated values. This separation—quote! for syntax, ToTokens for types—enables both readable macro code and extensible code generation systems. When building complex procedural macros, implementing ToTokens for your intermediate representations is often cleaner than constructing TokenStream objects directly, as it provides a uniform interface that integrates naturally with the entire quote and syn ecosystem.