Documentation interpolation

[clarfonthey]

You ever get so annoyed by something that you write up an RFC to fix it?

Basically expecting this to be completely ignored for one week minimum due to Rust Week, but this is something that always annoyed me and so I figured that I would try and take a stab at it.

Summary

Add the ability to use #{...} expressions in documentation comments for macro interpolation.

For example, #{$name} would be replaced with the result of stringify!($name) if used in documentation comments, and expressions like #{my_macro!($name)} also work.

Important

Since RFCs involve many conversations at once that can be difficult to follow, please use review comment threads on the text changes instead of direct comments on the RFC.

If you don't have a particular section of the RFC to comment on, you can click on the "Comment on this file" button on the top-right corner of the diff, to the right of the "Viewed" checkbox. This will create a separate thread even if others have commented on the file too.

Rendered

[tgross35]

Perhaps a prefixed foo( doc#{$bar} ) syntax would be a good option to (1) further reduce conflict chances, and (2) give a cue that it's documentation-related rather than something in the code.

View changes since the review

[clarfonthey]

Honestly, changing the syntax to #{...}, the syntax does kind of write itself…

Will have to ponder.

[tgross35]

As an alternative solution to solve the same problem, things would probably be less terrible if we had an inline concat syntax. Something like:

#[doc = concat2!("assert_eq!(({$SelfT}::MIN + 2).checked_sub(1), Some({$SelfT}), "::MIN + 1));")]
#[doc = concat2!("assert_eq!(({$SelfT}::MIN + 2).checked_sub(3), None);")]

Not that we couldn't have both :)

View changes since the review

[clarfonthey]

I actually do like this idea a lot, although I would like to still keep the doc comment syntax since I think that the misalignment between /// and #[doc = ...] really makes reading things harder.

General string interpolation for macros itself is something that would be worth exploring as worthwhile in its own right, and probably doesn't need to replicate the entire format_args! machinery to be useful.

[lebensterben]

even though I understand the intention of this PR, it still makes me feel like this new expression isn't just rustdoc only, but changing the transcribing of macro by example.

also, how would repetition be dealt with?

View changes since the review

[clarfonthey]

This is making me think that this could probably genuinely use a few more examples.

Repetition, to me, would be done exactly how it's done any other way:

$(
    /// ${$thing}.
    struct $thing;
)

Would work, if $thing is repeating at its current depth.

[lebensterben]

I'd like to propose an alternative to this new syntax.

Let's support fenced_code_attributes extension of Pandoc's Markdown

For code blocks, it would be one of the following equivalent:

```{.rust macro_interpolation=true}
assert_eq!(($SelfT::MIN + 2).checked_sub(3), None);
```

```rust {macro_interpolation=true}
assert_eq!(($SelfT::MIN + 2).checked_sub(3), None);
```

```{macro_interpolation=true}
assert_eq!(($SelfT::MIN + 2).checked_sub(3), None);
```

Here .rust is a class, and macro_interpolation=true is an attribute. (We may also allow named code block by using #id.)

The first syntax is the full syntax. The second one is a shorthand for code blocks in Pandoc's Markdown. And the third one is RUSTDOC's shorthand for defaulting code block language to Rust.

Similarly, for inline code, we can allow inline_code_attributes:

`assert_eq!(($SelfT::MIN + 2).checked_sub(3), None);`{.rust macro_interpolation=true}

(optionally allow .rust be omitted)

---

Now suppose for some reason, within a code block we have some items that start with $ sigil and we don't want it to be interpolated, but others do. We should introduce a way to escape interpolation, for example by escaping $.

---

It's worth noting that allowing macro interpolation everywhere could be dangerous. At least in short term I think we should only allow std and rust internal macros to be interpolated.

View changes since this review

[tmccombs]

Another option, which has better backwards compatibility is that we start with an attribute to opt in, then in a future edition make that the default (possibly with an attribute to opt out.

View changes since the review

[clarfonthey]

Right, this is kind of what I meant here, but I should reword to be more clear. What I meant by having the option to opt out is for forward compatibility, if you want to upgrade editions.

[kpreid]

Markdown consistently uses \ to remove the special meaning of characters, including in odd-looking cases such as 1\. to make a non-list-item. Therefore, I think the escaping syntax should be either of \#{ or #\{, instead of #{{, for consistency.

View changes since the review

[clarfonthey]

This is a fair point, although another thing worth considering is that this will also be interpolated in Rust code blocks where such syntax would appear a bit weird.

[kpreid]

An alternative I don't see considered here is to concatenate without also including the effect of stringify!. This would allow this interpolation syntax to also be used:

• to interpolate calls to macros that expand to strings, or
• to interpolate strings of documentation text that were passed into a macro by matching $(#[doc = $doc:literal] )* (these must not be stringify!()ed because that would quote and escape them)

which are other use cases where one currently must write #[doc = concat!(...)]). This is of course only a convenience syntax, and so it may not be worth supporting that use case, but I would like it to be considered explicitly. Searching my own code, I have a roughly equal number of cases where stringify! is needed and cases where it must be avoided.

View changes since the review

[clarfonthey]

So, I think I understand what you're trying to say here, but it would be helpful to have an example just to make sure. Because interpolating calls to macros definitely feels like it would require more than just what you're mentioning.

[kpreid]

interpolating calls to macros definitely feels like it would require more than just what you're mentioning

You can, today, write:

macro_rules! string_producing_macro {
    () => { "world" };
}

#[doc = concat!("Hello ", string_producing_macro!(), ".")]
pub struct SomeItem;

#[test]
fn test_documentation_string() {
    assert_eq!(string_producing_macro!(), "world");
}

This can be used to assert that certain documentation fragments are consistent with the behavior of the code, or to repeat a single block of text across multiple items’ documentation. (include_str!() can also be used in a similar way, but I wanted to emphasize that this is not just for already-prepared text but for any macro.)

What I’m asking for is to consider making the interpolation syntax not automatically stringify (or make a variant of it that does not stringify), so that one can write

/// Hello #{string_producing_macro!()}.

to the same effect as the concat!() version.

[clarfonthey]

Ah! I understand what you mean. Yes, I think that it would be reasonable to special-case string literals to just output their contents instead of always calling stringify.

[fmease]

#{...} expressions are rustdoc-only, so, they require no changes to the compiler, only rustdoc. However, they should apply to both /// comment syntax and #[doc = "comment"] syntax, since there should be no meaningful distinction between the two.

[…] #{...} cannot simply be replaced with equivalent #[doc = ...] attributes before documenting […]

The ///X→#[doc = "X"], /**X*/→#[doc="X"] desugaring is a proper user-observable part of the Rust language semantics. Consider this valid Rust program:

macro_rules! extract {
    (#[doc = $text:literal]) => { $text };
}

const _: &str = extract!(/**X*/);

That means the #{…} syntax you propose can be observed in its unexpanded form if they have the semantics you describe (assuming I've interpreted them correctly). That feels quite iffy to me.

View changes since the review

[fmease]

#{...} expressions are rustdoc-only, so, they require no changes to the compiler, only rustdoc. However, they should apply to both /// comment syntax and #[doc = "comment"] syntax, since there should be no meaningful distinction between the two.

[…] #{...} cannot simply be replaced with equivalent #[doc = ...] attributes before documenting, since this could cause entire files to fail to document due to invalid syntax.

Unless I misunderstood something, the semantics these two statements are trying to describe are impossible to implement:

As far as I can tell, the second quote is saying that /// #{...} does not expand to #[doc = concat!(" ", stringify!(...))] since that would lead to lexical/syntax errors during macro expansion if placeholder ... was lexically malformed; instead it would (continue to) expand to #[doc = " #{...}"] and rustdoc would then magically expand the #{...} when analyzing the contents of the expanded doc attributes later on (rendering it verbatim and issuing lint warnings if it's ill-formed) "similar to how nightly rustdoc can perform macro expansion on source code pages"¹.

However, if that were the case, rustdoc would need to be able to expand metavariables after expansion! That's an impossible ask! When encountering the string $x in #{$x}, no macro variable named $x will exist even if the doc comment / string literal comes from an expansion of "a macro that defines a macro metavariable called $x". MBE transcription happens "eagerly", it's a mere find&replace on macro transcribers!

Yes, rustdoc can probably find the corresponding macro from the source span but then what? It would need to "partially rerun" the macro somehow, taking into account macro repetitions (for e.g., $( ignore!($xs) /** #{$xs} */ )+).

---

Now, for the sake of argument, let's assume you say "okay, scrap this idea then; let's perform the #{...} expansion during macro expansion and accept lexical errors in #{...} being propagated". Implementation-wise that would mean modifying the compiler to provide hooks into macro expansion for rustdoc to use // implement the desired semantics if tcx.sess.opts.actually_rustdoc. Specification-wise it would mean that we end up with two distinct versions of Rust, the "real one" and the one as understood by rustdoc. That's not good. E.g., rustdoc would accept $( /**#{$xs}*/ )+ wheras rustc wouldn't (it can't "see" the $xs and thus the macro repetition is ill-defined); rustdoc would reject /**#{(}*/ wheras rustc wouldn't. Moreover, it would mean rustdoc has to inspect all string literals to look for #{...} (at that point in time, #[doc = "#{...}"] would just be a sequence of tokens, not an attribute, and we certainly don't want to speculatively try to parse attributes like that during expansion / look behind smh.), possible but undesirable(?).

Finally, we could say: Let's make #{...} part of the language then; that'd obviously require T-lang input and come with a lot more follow-up questions. It would also still mean that the #[doc = "#{...}"] idea can't be part of this (for the reason described in the paragraph above).

View changes since the review

Footnotes

1. Likely what "has its own tools for macro expansions" is referring to in "There are lots of methods to implement this, but since rustdoc has access to all the compiler internals and already has its own tools for macro expansion, this should be relatively easy." ↩

[notriddle]

We have our own tools for macro expansion? Other than running Boolean algebra on cfg() expressions, where are we doing that?

View changes since the review

[notriddle]

Wait, you’re talking about macro evaluation on the right-hand side of this equal sign?

#[doc=include_str!("foo")]

Sure, that might exist for rustdoc, but it’s in the compiler. If it fails to evaluate, your crate won’t build.

https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=14cafa595127cf01fa4673dead37c6ce

[fmease]

They might also be referring to the macro expansion on source code pages that's gated behind an unstable flag, not sure.

[notriddle]

Isn’t that feature, on its own, enough? If you need access to the result of evaluating a macro, you can assign it to a constant, then refer to that from the doc:

const FOO: &str = include_str!("foo.md");
/// #{FOO}
pub struct Foo;

Yes, it’s verbose, but it’s a lot better than doc=concat!(…), and I think we could implement it in the same pass as intra-doc link resolution (assuming we only let you point at a constant item, not write arbitrary inline expressions). And since it lets you transclude the results of running a const fn, it actually gives you new power, not just syntactic sugar.

View changes since the review

[kpreid]

And since it lets you transclude the results of running a const fn, it actually gives you new power, not just syntactic sugar.

Tangent: It would be useful to be able to embed the result of evaluating a const expression in documentation, for similar reasons to my previous comment about not stringify!()ing everything but going further: it would allow documentation to, for example, run a function in in order to discuss example outputs of that function, or (in some cases, and perhaps with added support for allocation in const) to generate tables/diagrams/figures from data that is in the code, without involving a proc-macro.

Furthermore, this could be a compatible extension to existing doc syntax without being tied to interpolation or having breaking syntax changes at all; simply make #[doc = some_constant_expr()] valid, which it currently is not. Generalize the current “must expand to a string literal” to “must evaluate to a string”, or even “must evaluate to a Display implementor” given const trait support.