Skip to content

simplify some proc_macro things #157271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cyrgani wants to merge 5 commits into
base: main
Choose a base branch
from
Open

simplify some proc_macro things #157271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
beb1d9e
remove some redundant closures
cyrgani May 30, 2026
bd45858
remove `ts_drop` bridge method
cyrgani May 30, 2026
1354363
simplify `Buffer`
cyrgani May 30, 2026
be8f3eb
rename `CustomDerive`s `trait_name` to `name`
cyrgani May 31, 2026
750ea38
set `issue = none` for `proc_macro_internals`
cyrgani Jun 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions compiler/rustc_expand/src/proc_macro_server.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,10 +556,6 @@ impl server::Server for Rustc<'_, '_> {
diag.emit();
}

fn ts_drop(&mut self, stream: Self::TokenStream) {
drop(stream);
}
Copy link
Copy Markdown
Member

@bjorn3 bjorn3 Jun 2, 2026 •
edited by rustbot
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this cause all tokenstreams to be unnecessarily retained on the server side until the end of the proc macro invocation even if the client drops it earlier?

View changes since the review

Copy link
Copy Markdown
Contributor

@petrochenkov petrochenkov Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the motivation is missing for this change now - what did this drop do, what happens now when it's removed, why it's good.

Copy link
Copy Markdown
Contributor Author

@cyrgani cyrgani Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason for this change is that currently all TokenStream implementations are wrapped in an Arc or Rc and I would not expect that to change. So none of them need anything special happening to drop them.
It seemed wasteful to me to have to go through the bridge just to drop the value considering this and the assumption that in most of these cases, dropping a token stream would just decrease the Arcs refcount by 1 without actually freeing the underlying value already. So I expected the perf gain from no longer going through the bridge to drop them to outweigh the cost of storing them a little longer. But I may very well be mistaken here.

Copy link
Copy Markdown
Member

@bjorn3 bjorn3 Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server side needs to be notified somehow that the client side dropped the TokenStream. Otherwise it will stick around in the OwnedStore until the proc macro finishes. This notifying is done by passing it to ts_drop.


fn ts_clone(&mut self, stream: &Self::TokenStream) -> Self::TokenStream {
stream.clone()
}
Expand Down
4 changes: 2 additions & 2 deletions compiler/rustc_metadata/src/rmeta/decoder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,11 +1047,11 @@ impl CrateMetadata {

fn load_proc_macro<'tcx>(&self, tcx: TyCtxt<'tcx>, id: DefIndex) -> SyntaxExtension {
let (name, kind, helper_attrs) = match *self.raw_proc_macro(tcx, id) {
ProcMacro::CustomDerive { trait_name, attributes, client } => {
ProcMacro::CustomDerive { name, attributes, client } => {
let helper_attrs =
attributes.iter().cloned().map(Symbol::intern).collect::<Vec<_>>();
(
trait_name,
name,
SyntaxExtensionKind::Derive(Arc::new(DeriveProcMacro { client })),
helper_attrs,
)
Expand Down
32 changes: 2 additions & 30 deletions library/proc_macro/src/bridge/buffer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
//! Buffer management for same-process client<->server communication.

use std::io::{self, Write};
use std::mem::{self, ManuallyDrop};
use std::ops::{Deref, DerefMut};
use std::ops::Deref;
use std::slice;

#[repr(C)]
Expand Down Expand Up @@ -32,13 +31,6 @@ impl Deref for Buffer {
}
}

impl DerefMut for Buffer {
#[inline]
fn deref_mut(&mut self) -> &mut [u8] {
unsafe { slice::from_raw_parts_mut(self.data, self.len) }
}
}

impl Buffer {
#[inline]
pub(super) fn new() -> Self {
Expand Down Expand Up @@ -99,25 +91,6 @@ impl Buffer {
}
}

impl Write for Buffer {
#[inline]
fn write(&mut self, xs: &[u8]) -> io::Result<usize> {
self.extend_from_slice(xs);
Ok(xs.len())
}

#[inline]
fn write_all(&mut self, xs: &[u8]) -> io::Result<()> {
self.extend_from_slice(xs);
Ok(())
}

#[inline]
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}

impl Drop for Buffer {
#[inline]
fn drop(&mut self) {
Expand All @@ -128,8 +101,7 @@ impl Drop for Buffer {

impl From<Vec<u8>> for Buffer {
fn from(v: Vec<u8>) -> Self {
let mut v = ManuallyDrop::new(v);
let (data, len, capacity) = (v.as_mut_ptr(), v.len(), v.capacity());
let (data, len, capacity) = v.into_raw_parts();

// This utility function is nested in here because it can *only*
// be safely called on `Buffer`s created by *this* `proc_macro`.
Expand Down
21 changes: 7 additions & 14 deletions library/proc_macro/src/bridge/client.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,6 @@ pub(crate) struct TokenStream {
impl !Send for TokenStream {}
impl !Sync for TokenStream {}

// Forward `Drop::drop` to the inherent `drop` method.
impl Drop for TokenStream {
fn drop(&mut self) {
Methods::ts_drop(TokenStream { handle: self.handle });
}
}

impl<S> Encode<S> for TokenStream {
fn encode(self, w: &mut Buffer, s: &mut S) {
mem::ManuallyDrop::new(self).handle.encode(w, s);
Expand Down Expand Up @@ -283,7 +276,7 @@ impl Client<crate::TokenStream, crate::TokenStream> {
pub const fn expand1(f: impl Fn(crate::TokenStream) -> crate::TokenStream + Copy) -> Self {
Client {
run: super::selfless_reify::reify_to_extern_c_fn_hrt_bridge(move |bridge| {
run_client(bridge, |input| f(input))
run_client(bridge, f)
}),
_marker: PhantomData,
}
Expand All @@ -307,7 +300,7 @@ impl Client<(crate::TokenStream, crate::TokenStream), crate::TokenStream> {
#[derive(Copy, Clone)]
pub enum ProcMacro {
CustomDerive {
trait_name: &'static str,
name: &'static str,
attributes: &'static [&'static str],
client: Client<crate::TokenStream, crate::TokenStream>,
},
Expand All @@ -326,18 +319,18 @@ pub enum ProcMacro {
impl ProcMacro {
pub fn name(&self) -> &'static str {
match self {
ProcMacro::CustomDerive { trait_name, .. } => trait_name,
ProcMacro::Attr { name, .. } => name,
ProcMacro::Bang { name, .. } => name,
ProcMacro::CustomDerive { name, .. }
| ProcMacro::Attr { name, .. }
| ProcMacro::Bang { name, .. } => name,
}
}

pub const fn custom_derive(
trait_name: &'static str,
name: &'static str,
attributes: &'static [&'static str],
expand: impl Fn(crate::TokenStream) -> crate::TokenStream + Copy,
) -> Self {
ProcMacro::CustomDerive { trait_name, attributes, client: Client::expand1(expand) }
ProcMacro::CustomDerive { name, attributes, client: Client::expand1(expand) }
}

pub const fn attr(
Expand Down
1 change: 0 additions & 1 deletion library/proc_macro/src/bridge/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ macro_rules! with_api {
fn literal_from_str(s: &str) -> Result<Literal<$Span, $Symbol>, String>;
fn emit_diagnostic(diagnostic: Diagnostic<$Span>);

fn ts_drop(stream: $TokenStream);
fn ts_clone(stream: &$TokenStream) -> $TokenStream;
fn ts_is_empty(stream: &$TokenStream) -> bool;
fn ts_expand_expr(stream: &$TokenStream) -> Result<$TokenStream, ()>;
Expand Down
3 changes: 1 addition & 2 deletions library/proc_macro/src/bridge/rpc.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
//! Serialization for client-server communication.

use std::any::Any;
use std::io::Write;
use std::num::NonZero;

use super::buffer::Buffer;
Expand Down Expand Up @@ -192,7 +191,7 @@ impl<S> Encode<S> for &str {
fn encode(self, w: &mut Buffer, s: &mut S) {
let bytes = self.as_bytes();
bytes.len().encode(w, s);
w.write_all(bytes).unwrap();
w.extend_from_slice(bytes);
}
}

Expand Down
14 changes: 6 additions & 8 deletions library/proc_macro/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
#![warn(unreachable_pub)]
#![deny(unsafe_op_in_unsafe_fn)]

#[unstable(feature = "proc_macro_internals", issue = "27812")]
#[unstable(feature = "proc_macro_internals", issue = "none")]
#[doc(hidden)]
pub mod bridge;

Expand Down Expand Up @@ -165,7 +165,7 @@ impl TokenStream {
/// Checks if this `TokenStream` is empty.
#[stable(feature = "proc_macro_lib2", since = "1.29.0")]
pub fn is_empty(&self) -> bool {
self.0.as_ref().map(|h| BridgeMethods::ts_is_empty(h)).unwrap_or(true)
self.0.as_ref().map(BridgeMethods::ts_is_empty).unwrap_or(true)
}

/// Parses this `TokenStream` as an expression and attempts to expand any
Expand Down Expand Up @@ -444,9 +444,7 @@ pub mod token_stream {
type IntoIter = IntoIter;

fn into_iter(self) -> IntoIter {
IntoIter(
self.0.map(|v| BridgeMethods::ts_into_trees(v)).unwrap_or_default().into_iter(),
)
IntoIter(self.0.map(BridgeMethods::ts_into_trees).unwrap_or_default().into_iter())
}
}
}
Expand All @@ -464,7 +462,7 @@ pub macro quote($($t:tt)*) {
/* compiler built-in */
}

#[unstable(feature = "proc_macro_internals", issue = "27812")]
#[unstable(feature = "proc_macro_internals", issue = "none")]
#[doc(hidden)]
mod quote;

Expand Down Expand Up @@ -624,14 +622,14 @@ impl Span {

// Used by the implementation of `Span::quote`
#[doc(hidden)]
#[unstable(feature = "proc_macro_internals", issue = "27812")]
#[unstable(feature = "proc_macro_internals", issue = "none")]
pub fn save_span(&self) -> usize {
BridgeMethods::span_save_span(self.0)
}

// Used by the implementation of `Span::quote`
#[doc(hidden)]
#[unstable(feature = "proc_macro_internals", issue = "27812")]
#[unstable(feature = "proc_macro_internals", issue = "none")]
pub fn recover_proc_macro_span(id: usize) -> Span {
Span(BridgeMethods::span_recover_proc_macro_span(id))
}
Expand Down
8 changes: 4 additions & 4 deletions src/tools/rust-analyzer/crates/proc-macro-srv/src/dylib/proc_macros.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ impl ProcMacros {

for proc_macro in &self.0 {
match proc_macro {
bridge::client::ProcMacro::CustomDerive { trait_name, client, .. }
if *trait_name == macro_name =>
bridge::client::ProcMacro::CustomDerive { name, client, .. }
if *name == macro_name =>
{
let res = client.run(
&bridge::server::SAME_THREAD,
Expand Down Expand Up @@ -65,8 +65,8 @@ impl ProcMacros {

pub(crate) fn list_macros(&self) -> impl Iterator<Item = (&str, ProcMacroKind)> {
self.0.iter().map(|proc_macro| match *proc_macro {
bridge::client::ProcMacro::CustomDerive { trait_name, .. } => {
(trait_name, ProcMacroKind::CustomDerive)
bridge::client::ProcMacro::CustomDerive { name, .. } => {
(name, ProcMacroKind::CustomDerive)
}
bridge::client::ProcMacro::Bang { name, .. } => (name, ProcMacroKind::Bang),
bridge::client::ProcMacro::Attr { name, .. } => (name, ProcMacroKind::Attr),
Expand Down
4 changes: 0 additions & 4 deletions src/tools/rust-analyzer/crates/proc-macro-srv/src/server_impl/rust_analyzer_span.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,6 @@ impl server::Server for RaSpanServer<'_> {
// FIXME handle diagnostic
}

fn ts_drop(&mut self, stream: Self::TokenStream) {
drop(stream);
}

fn ts_clone(&mut self, stream: &Self::TokenStream) -> Self::TokenStream {
stream.clone()
}
Expand Down
4 changes: 0 additions & 4 deletions src/tools/rust-analyzer/crates/proc-macro-srv/src/server_impl/token_id.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,6 @@ impl server::Server for SpanIdServer<'_> {

fn emit_diagnostic(&mut self, _: Diagnostic<Self::Span>) {}

fn ts_drop(&mut self, stream: Self::TokenStream) {
drop(stream);
}

fn ts_clone(&mut self, stream: &Self::TokenStream) -> Self::TokenStream {
stream.clone()
}
Expand Down
Loading