Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions crates/hir-def/src/macro_expansion_tests/mbe/regression.rs
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,42 @@ fn main() {
);
}

#[test]
fn ty_fragment_followed_by_expr() {
check(
r#"
macro_rules! a {
($t:tt) => {};
}

macro_rules! b {
($t:ty) => {
a!($t);
};
}

fn main() {
b!(&'static str);
}
"#,
expect![[r#"
macro_rules! a {
($t:tt) => {};
}

macro_rules! b {
($t:ty) => {
a!($t);
};
}

fn main() {
a!(&'static str);;
}
"#]],
);
}

#[test]
fn test_winapi_struct() {
// from https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/macros.rs#L366
Expand Down
13 changes: 11 additions & 2 deletions crates/mbe/src/expander.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,10 @@ enum Fragment<'a> {
#[default]
Empty,
/// token fragments are just copy-pasted into the output
Tokens(tt::TokenTreesView<'a, Span>),
Tokens {
tree: tt::TokenTreesView<'a, Span>,
origin: TokensOrigin,
},
Comment on lines 130 to +134
Copy link

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation comment for the Tokens variant should be updated to explain the new origin field. The current comment "token fragments are just copy-pasted into the output" is incomplete. Consider updating it to:

/// Token fragments are transcribed into the output. The `origin` field tracks whether
/// these tokens come from raw input (ident/tt/lifetime/literal metavars) or parsed AST
/// (ty/pat/stmt/block/etc metavars). AST-derived fragments preserve their structure
/// during transcription by not stripping invisible delimiters.

This provides context for the new field and explains why it matters.

Copilot uses AI. Check for mistakes.
/// Expr ast fragments are surrounded with `()` on transcription to preserve precedence.
/// Note that this impl is different from the one currently in `rustc` --
/// `rustc` doesn't translate fragments into token trees at all.
Expand Down Expand Up @@ -156,10 +159,16 @@ impl Fragment<'_> {
fn is_empty(&self) -> bool {
match self {
Fragment::Empty => true,
Fragment::Tokens(it) => it.len() == 0,
Fragment::Tokens { tree, .. } => tree.len() == 0,
Fragment::Expr(it) => it.len() == 0,
Fragment::Path(it) => it.len() == 0,
Fragment::TokensOwned(it) => it.0.is_empty(),
}
}
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum TokensOrigin {
Raw,
Ast,
}
Comment on lines +170 to +174
Copy link

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The TokensOrigin enum should have documentation explaining the distinction between Raw and Ast variants. Consider adding a doc comment like:

/// Tracks whether token fragments come from raw tokens or parsed AST.
/// This distinction is important during transcription: AST-derived fragments
/// should not have their invisible delimiters stripped to preserve their
/// structure as complete syntactic units.

This would help future maintainers understand the purpose of this enum and when to use each variant.

Copilot uses AI. Check for mistakes.
27 changes: 16 additions & 11 deletions crates/mbe/src/expander/matcher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ use tt::{

use crate::{
ExpandError, ExpandErrorKind, MetaTemplate, ValueResult,
expander::{Binding, Bindings, ExpandResult, Fragment},
expander::{Binding, Bindings, ExpandResult, Fragment, TokensOrigin},
expect_fragment,
parser::{ExprKind, MetaVarKind, Op, RepeatKind, Separator},
};
Expand Down Expand Up @@ -842,18 +842,23 @@ fn match_meta_var<'t>(
}
.err();
let tt_result = input.from_savepoint(savepoint);
return ValueResult { value: Fragment::Tokens(tt_result), err };
return ValueResult {
value: Fragment::Tokens { tree: tt_result, origin: TokensOrigin::Raw },
err,
};
}
MetaVarKind::Ty => parser::PrefixEntryPoint::Ty,
MetaVarKind::Pat => parser::PrefixEntryPoint::PatTop,
MetaVarKind::PatParam => parser::PrefixEntryPoint::Pat,
MetaVarKind::Stmt => parser::PrefixEntryPoint::Stmt,
MetaVarKind::Block => parser::PrefixEntryPoint::Block,
MetaVarKind::Meta => parser::PrefixEntryPoint::MetaItem,
MetaVarKind::Item => parser::PrefixEntryPoint::Item,
MetaVarKind::Vis => parser::PrefixEntryPoint::Vis,
MetaVarKind::Ty => (parser::PrefixEntryPoint::Ty, TokensOrigin::Ast),
MetaVarKind::Pat => (parser::PrefixEntryPoint::PatTop, TokensOrigin::Ast),
MetaVarKind::PatParam => (parser::PrefixEntryPoint::Pat, TokensOrigin::Ast),
MetaVarKind::Stmt => (parser::PrefixEntryPoint::Stmt, TokensOrigin::Ast),
MetaVarKind::Block => (parser::PrefixEntryPoint::Block, TokensOrigin::Ast),
MetaVarKind::Meta => (parser::PrefixEntryPoint::MetaItem, TokensOrigin::Ast),
MetaVarKind::Item => (parser::PrefixEntryPoint::Item, TokensOrigin::Ast),
MetaVarKind::Vis => (parser::PrefixEntryPoint::Vis, TokensOrigin::Ast),
};
expect_fragment(db, input, fragment, delim_span).map(Fragment::Tokens)
let (entry_point, origin) = fragment;
expect_fragment(db, input, entry_point, delim_span)
.map(|tree| Fragment::Tokens { tree, origin })
}

fn collect_vars(collector_fun: &mut impl FnMut(Symbol), pattern: &MetaTemplate) {
Expand Down
11 changes: 9 additions & 2 deletions crates/mbe/src/expander/transcriber.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ use intern::{Symbol, sym};
use span::{Edition, Span};
use tt::{Delimiter, TopSubtreeBuilder, iter::TtElement};

use super::TokensOrigin;
use crate::{
ExpandError, ExpandErrorKind, ExpandResult, MetaTemplate,
expander::{Binding, Bindings, Fragment},
Expand Down Expand Up @@ -313,7 +314,7 @@ fn expand_subtree(
}
};
let values = match &var_value {
Fragment::Tokens(tokens) => {
Fragment::Tokens { tree: tokens, .. } => {
let mut iter = tokens.iter();
(iter.next(), iter.next())
}
Expand Down Expand Up @@ -393,7 +394,13 @@ fn expand_var(
// rustc spacing is not like ours. Ours is like proc macros', it dictates how puncts will actually be joined.
// rustc uses them mostly for pretty printing. So we have to deviate a bit from what rustc does here.
// Basically, a metavariable can never be joined with whatever after it.
Fragment::Tokens(tt) => builder.extend_with_tt_alone(tt.strip_invisible()),
Fragment::Tokens { tree, origin } => {
let view = match origin {
TokensOrigin::Raw => tree.strip_invisible(),
TokensOrigin::Ast => tree,
};
builder.extend_with_tt_alone(view);
}
Fragment::TokensOwned(tt) => {
builder.extend_with_tt_alone(tt.view().strip_invisible())
}
Expand Down