What is the purpose of `quote::ToTokens::to_tokens` for custom token stream generation?

quote::ToTokens::to_tokens is the fundamental trait method that enables types to contribute their Rust token representation to a TokenStream, allowing custom types to participate in procedural macro code generation alongside built-in syntax elements. The quote! macro works by calling to_tokens on every interpolated value, which is why you can write quote! { let x = #value; } with value being a TokenStream, a String, a proc_macro2::Literal, or any type implementing ToTokens. Implementing ToTokens for your own types lets them seamlessly integrate into quote! invocations, generating custom token streams that represent complex data structures, derive implementations, or domain-specific code patterns. The method receives a &mut TokenStream and appends tokens to it rather than returning a new TokenStream, enabling efficient composition without allocation overhead.

Basic ToTokens Implementation

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
struct Variable {
    name: String,
    ty: String,
}
 
impl ToTokens for Variable {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        
        // Append tokens representing this variable
        tokens.append_all(quote! {
            let #name: #ty;
        });
    }
}
 
fn main() {
    let var = Variable {
        name: "count".to_string(),
        ty: "i32".to_string(),
    };
    
    // Now we can use Variable directly in quote!
    let tokens = quote! {
        fn example() {
            #var
        }
    };
    
    println!("{}", tokens);
}

Implementing ToTokens lets your types work directly in quote! macros.

TokenStream Appending Pattern

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
struct StructDefinition {
    name: String,
    fields: Vec<(String, String)>,
}
 
impl ToTokens for StructDefinition {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let fields = self.fields.iter().map(|(field_name, field_ty)| {
            quote! { #field_name: #field_ty, }
        });
        
        // Append the struct definition
        tokens.append_all(quote! {
            struct #name {
                #(#fields)*
            }
        });
    }
}
 
fn main() {
    let def = StructDefinition {
        name: "Person".to_string(),
        fields: vec
![
            ("name".to_string(), "String".to_string()),
            ("age".to_string(), "u32".to_string()),
        ],
    };
    
    let tokens = quote! {
        #def
    };
    
    println!("{}", tokens);
}

append_all adds multiple tokens; the method modifies tokens in place.

Built-in ToTokens Implementations

use quote::quote;
 
fn main() {
    // String implements ToTokens (as string literal)
    let name = String::from("value");
    let tokens = quote! { let s = #name; };
    println!("String: {}", tokens);
    
    // &str implements ToTokens (as string literal)
    let literal = "hello";
    let tokens = quote! { let s = #literal; };
    println!("&str: {}", tokens);
    
    // i32/u32/etc implement ToTokens (as numeric literal)
    let count: i32 = 42;
    let tokens = quote! { const COUNT: i32 = #count; };
    println!("i32: {}", tokens);
    
    // bool implements ToTokens
    let flag = true;
    let tokens = quote! { const FLAG: bool = #flag; };
    println!("bool: {}", tokens);
    
    // TokenStream implements ToTokens (passthrough)
    let inner = quote! { 1 + 2 };
    let tokens = quote! { let result = #inner; };
    println!("TokenStream: {}", tokens);
}

Many standard types already implement ToTokens for common use cases.

Custom Types with ToTokens

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::{TokenStream, Ident, Span};
 
struct Field {
    name: Ident,
    ty: Ident,
}
 
impl ToTokens for Field {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        
        tokens.append_all(quote! {
            #name: #ty,
        });
    }
}
 
fn main() {
    let fields = vec
![
        Field {
            name: Ident::new("id", Span::call_site()),
            ty: Ident::new("u64", Span::call_site()),
        },
        Field {
            name: Ident::new("name", Span::call_site()),
            ty: Ident::new("String", Span::call_site()),
        },
    ];
    
    // Vec implements ToTokens using repetition
    let tokens = quote! {
        struct Record {
            #(#fields)*
        }
    };
    
    println!("{}", tokens);
}

Collections of ToTokens types work with quote! repetition.

ToTokens for Derive Macros

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::{TokenStream, Ident, Span};
use syn::{parse_quote, DeriveInput};
 
// Example: A custom derive implementation helper
struct DerivedTrait {
    trait_name: Ident,
    impl_body: TokenStream,
}
 
impl ToTokens for DerivedTrait {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let trait_name = &self.trait_name;
        let body = &self.impl_body;
        
        tokens.append_all(quote! {
            impl #trait_name for Target {
                #body
            }
        });
    }
}
 
fn main() {
    let derived = DerivedTrait {
        trait_name: Ident::new("Debug", Span::call_site()),
        impl_body: quote! {
            fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
                write!(f, "Target")
            }
        },
    };
    
    let tokens = quote! { #derived };
    println!("{}", tokens);
}

ToTokens implementations are essential for procedural macro code generation.

ToTokens vs to_token_stream

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
struct Example {
    value: i32,
}
 
impl ToTokens for Example {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let value = self.value;
        // Append to existing TokenStream
        tokens.append_all(quote! { #value });
    }
}
 
impl Example {
    // Alternative: return a new TokenStream
    fn to_token_stream(&self) -> TokenStream {
        let value = self.value;
        quote! { #value }
    }
}
 
fn main() {
    let ex = Example { value: 42 };
    
    // Using ToTokens
    let mut tokens = quote! { let x = };
    ex.to_tokens(&mut tokens);
    tokens.extend(quote! { ; });
    println!("ToTokens: {}", tokens);
    
    // Using to_token_stream
    let tokens = quote! { let x = #ex; };
    println!("In quote: {}", tokens);
    
    // The quote! macro uses ToTokens internally
}

to_tokens appends; to_token_stream returns a new TokenStream.

Nested Token Composition

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::{TokenStream, Ident, Span};
 
struct Attribute {
    name: Ident,
    value: String,
}
 
impl ToTokens for Attribute {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let value = &self.value;
        tokens.append_all(quote! { #[#name = #value] });
    }
}
 
struct Field {
    attrs: Vec<Attribute>,
    name: Ident,
    ty: Ident,
}
 
impl ToTokens for Field {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let attrs = &self.attrs;
        let name = &self.name;
        let ty = &self.ty;
        
        tokens.append_all(quote! {
            #(#attrs)*
            #name: #ty,
        });
    }
}
 
fn main() {
    let field = Field {
        attrs: vec
![
            Attribute {
                name: Ident::new("doc", Span::call_site()),
                value: "User name".to_string(),
            },
        ],
        name: Ident::new("username", Span::call_site()),
        ty: Ident::new("String", Span::call_site()),
    };
    
    let tokens = quote! {
        struct User {
            #field
        }
    };
    
    println!("{}", tokens);
}

Complex nested structures compose naturally through ToTokens.

Conditional Token Generation

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
struct OptionalField {
    name: String,
    ty: String,
    is_optional: bool,
}
 
impl ToTokens for OptionalField {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        let ty = &self.ty;
        
        if self.is_optional {
            // Generate Option<T> type
            tokens.append_all(quote! {
                #name: Option<#ty>,
            });
        } else {
            // Generate bare type
            tokens.append_all(quote! {
                #name: #ty,
            });
        }
    }
}
 
fn main() {
    let fields = vec
![
        OptionalField {
            name: "id".to_string(),
            ty: "u64".to_string(),
            is_optional: false,
        },
        OptionalField {
            name: "nickname".to_string(),
            ty: "String".to_string(),
            is_optional: true,
        },
    ];
    
    let tokens = quote! {
        struct Config {
            #(#fields)*
        }
    };
    
    println!("{}", tokens);
}

ToTokens implementations can include conditional logic for token generation.

ToTokens for Enums

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::{TokenStream, Ident, Span};
 
enum TypeRef {
    Simple(Ident),
    Generic { base: Ident, args: Vec<TypeRef> },
    Array { elem: Box<TypeRef>, size: usize },
}
 
impl ToTokens for TypeRef {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        match self {
            TypeRef::Simple(ident) => {
                ident.to_tokens(tokens);
            }
            TypeRef::Generic { base, args } => {
                tokens.append_all(quote! {
                    #base<#(#args),*>
                });
            }
            TypeRef::Array { elem, size } => {
                tokens.append_all(quote! {
                    [#elem; #size]
                });
            }
        }
    }
}
 
fn main() {
    let types = vec
![
        TypeRef::Simple(Ident::new("i32", Span::call_site())),
        TypeRef::Generic {
            base: Ident::new("Vec", Span::call_site()),
            args: vec
![TypeRef::Simple(Ident::new("String", Span::call_site()))],
        },
        TypeRef::Array {
            elem: Box::new(TypeRef::Simple(Ident::new("u8", Span::call_site()))),
            size: 32,
        },
    ];
    
    let tokens = quote! {
        fn example(#(#types),*) {}
    };
    
    println!("{}", tokens);
}

Enum variants can generate different token patterns.

syn Types and ToTokens

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
use syn::{Ident, parse_quote, Type, Item};
 
fn main() {
    // syn types implement ToTokens
    let ident: Ident = parse_quote! { my_function };
    let ty: Type = parse_quote! { Vec<String> };
    
    let tokens = quote! {
        fn #ident() -> #ty {
            Vec::new()
        }
    };
    println!("Function: {}", tokens);
    
    // Complete AST nodes can be converted back to tokens
    let item: Item = parse_quote! {
        struct Point {
            x: f64,
            y: f64,
        }
    };
    
    let tokens = quote! { #item };
    println!("Struct: {}", tokens);
}

All syn AST types implement ToTokens, enabling round-trip parsing.

Custom Token Generation with proc_macro2

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::{TokenStream, TokenTree, Ident, Literal, Span, Punct, Spacing};
 
struct CustomTokens;
 
impl ToTokens for CustomTokens {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        // Manually construct tokens
        let ident = Ident::new("custom", Span::call_site());
        let punct = Punct::new(':', Spacing::Joint);
        let punct2 = Punct::new(':', Spacing::Alone);
        let literal = Literal::i32_unsuffixed(42);
        
        tokens.append(ident);
        tokens.append(punct);
        tokens.append(punct2);
        tokens.append(literal);
    }
}
 
fn main() {
    let custom = CustomTokens;
    
    let tokens = quote! {
        let value = #custom;
    };
    
    println!("{}", tokens);
}

For fine-grained control, manually construct TokenTree values.

ToTokens for Trait Bounds

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::{TokenStream, Ident, Span};
 
struct GenericParam {
    name: Ident,
    bounds: Vec<Ident>,
}
 
impl ToTokens for GenericParam {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        
        if self.bounds.is_empty() {
            tokens.append_all(quote! { #name });
        } else {
            let bounds = &self.bounds;
            tokens.append_all(quote! { #name: #(#bounds)+* });
        }
    }
}
 
fn main() {
    let params = vec
![
        GenericParam {
            name: Ident::new("T", Span::call_site()),
            bounds: vec
![Ident::new("Clone", Span::call_site()), Ident::new("Debug", Span::call_site())],
        },
        GenericParam {
            name: Ident::new("U", Span::call_site()),
            bounds: vec
![],
        },
    ];
    
    let tokens = quote! {
        struct Container<#(#params),*> {
            _inner: T,
        }
    };
    
    println!("{}", tokens);
}

Complex generics can be represented with custom ToTokens implementations.

Efficiency: Appending vs Returning

use quote::{quote, ToTokens, TokenStreamExt};
use proc_macro2::TokenStream;
 
struct Efficient {
    parts: Vec<String>,
}
 
impl ToTokens for Efficient {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        // Efficient: append each part to the same TokenStream
        for part in &self.parts {
            tokens.append_all(quote! { #part });
        }
    }
}
 
impl Efficient {
    // Less efficient: creates intermediate TokenStreams
    fn to_token_stream(&self) -> TokenStream {
        let mut result = TokenStream::new();
        for part in &self.parts {
            result.extend(quote! { #part });
        }
        result
    }
}
 
fn main() {
    let eff = Efficient {
        parts: vec
!["a".to_string(), "b".to_string(), "c".to_string()],
    };
    
    // Using ToTokens (append pattern)
    let mut tokens = TokenStream::new();
    eff.to_tokens(&mut tokens);
    println!("ToTokens: {}", tokens);
}

Appending to an existing TokenStream avoids intermediate allocations.

ToTokens with proc_macro vs proc_macro2

// In a procedural macro crate
use proc_macro::TokenStream as ProcTokenStream;
use proc_macro2::TokenStream;
use quote::{quote, ToTokens};
 
// Convert proc_macro2 to proc_macro
fn into_proc(tokens: TokenStream) -> ProcTokenStream {
    tokens.into()
}
 
// Convert proc_macro to proc_macro2
fn from_proc(tokens: ProcTokenStream) -> TokenStream {
    TokenStream::from(tokens)
}
 
// ToTokens works with proc_macro2
// Use .into() to convert to proc_macro for return

Procedural macros use proc_macro::TokenStream; quote uses proc_macro2.

Debug Token Inspection

use quote::{quote, ToTokens};
use proc_macro2::TokenStream;
 
fn inspect_tokens(tokens: &TokenStream) {
    println!("Token stream: {}", tokens);
    println!("Raw tokens: {:?}", tokens);
    
    for token in tokens.clone().into_iter() {
        println!("  Token: {:?}", token);
    }
}
 
struct DebugExample {
    name: String,
}
 
impl ToTokens for DebugExample {
    fn to_tokens(&self, tokens: &mut TokenStream) {
        let name = &self.name;
        tokens.extend(quote! { #name });
    }
}
 
fn main() {
    let ex = DebugExample {
        name: "test_value".to_string(),
    };
    
    let tokens = quote! {
        let #ex = 42;
    };
    
    inspect_tokens(&tokens);
}

Inspecting tokens helps debug macro expansion.

Synthesis

ToTokens method signature:

trait ToTokens {
    fn to_tokens(&self, tokens: &mut TokenStream);
}

Common implementations:

| Type | Token Output | |------|--------------| | String, &str | String literal | | i32, u64, etc. | Numeric literal | | bool | true or false | | TokenStream | Passed through | | Ident | Identifier token | | Vec<T: ToTokens> | Each element's tokens |

When to implement ToTokens:

| Scenario | Reason | |----------|--------| | Custom derive output | Generate impl blocks | | Code generation helpers | Compose token fragments | | Domain-specific types | Generate matching Rust syntax | | Conditional code gen | Logic-driven token production | | AST manipulation | Transform syn types back to tokens |

Key insight: ToTokens is the foundational abstraction that makes quote! work. When you write quote! { let x = #value; }, the macro expands to code that calls value.to_tokens(&mut tokens) for each interpolated variable. The mutable append pattern—passing &mut TokenStream rather than returning TokenStream—is deliberate: it enables efficient composition where multiple token producers write to the same buffer without intermediate allocations. This matters deeply in procedural macros, where you might compose hundreds of token fragments into a single derive implementation. The trait's simplicity belies its power: by implementing to_tokens, your types become first-class citizens in the quote! ecosystem, seamlessly interoperating with syn AST types, standard Rust types, and other custom implementations. The pattern of "append to a mutable buffer" also enables natural composition—implement ToTokens for small pieces (fields, attributes, bounds), and they automatically compose into larger structures (structs, impls, functions) without any extra plumbing. This is why the quote/syn ecosystem is the standard for Rust procedural macros: ToTokens provides a uniform interface for code generation that just works.

Loading page…

What is the purpose of quote::ToTokens::to_tokens for custom token stream generation?

Basic ToTokens Implementation

TokenStream Appending Pattern

Built-in ToTokens Implementations

Custom Types with ToTokens

ToTokens for Derive Macros

ToTokens vs to_token_stream

Nested Token Composition

Conditional Token Generation

ToTokens for Enums

syn Types and ToTokens

Custom Token Generation with proc_macro2

ToTokens for Trait Bounds

Efficiency: Appending vs Returning

ToTokens with proc_macro vs proc_macro2

Debug Token Inspection

Synthesis

What is the purpose of quote::ToTokens::to_tokens for custom token stream generation?

Basic ToTokens Implementation

TokenStream Appending Pattern

Built-in ToTokens Implementations

Custom Types with ToTokens

ToTokens for Derive Macros

ToTokens vs to_token_stream

Nested Token Composition

Conditional Token Generation

ToTokens for Enums

syn Types and ToTokens

Custom Token Generation with proc_macro2

ToTokens for Trait Bounds

Efficiency: Appending vs Returning

ToTokens with proc_macro vs proc_macro2

Debug Token Inspection

Synthesis

What is the purpose of `quote::ToTokens::to_tokens` for custom token stream generation?

What is the purpose of `quote::ToTokens::to_tokens` for custom token stream generation?