feat: support Lua and JavaScript extensions#1196
Conversation
✨ Highlights
🧾 Changes by Scope
🔝 Top Files
|
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
ef7ea6b to
2156b1c
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1196 +/- ##
========================================
Coverage 82.12% 82.12%
========================================
Files 33 33
Lines 3149 3149
Branches 734 734
========================================
Hits 2586 2586
Misses 387 387
Partials 176 176
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
An automated preview of the documentation is available at https://1196.mrdocs.prtest2.cppalliance.org/index.html If more commits are pushed to the pull request, the docs will rebuild at the same URL. 2026-05-19 16:19:08 UTC |
gennaroprota
changed the title
gennaroprota
changed the title
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
2 times, most recently
from
63f542c to
8c6f453
Compare
This mirrors the existing JS helpers for Lua. *.lua files placed in an
addon's generator/{common|<ext>}/helpers/ directory are auto-registered
as Handlebars helpers; files whose name starts with '_' run first as
utility scripts. Two golden fixtures (lua-helper/, lua-helper-layering/)
mirror their JS counterparts and cover the `addons-supplemental`
override.
Incidental fixes to issues uncovered by this patch:
- Added a qualification to `MRDOCS_TRY` / `MRDOCS_CHECK_*` /
`MRDOCS_CHECK_OR_*` / `MRDOCS_CHECK_OR_CONTINUE` to make them work
with nested namespaces named `detail`.
- Dropped onelua.c and ltests.c from the Lua build patch, because the
former defines `main`, which conflicted with our `main`, and the
latter is test scaffolding which shouldn't ship in a library build.
- Added `extern "C"` around the Lua includes.
The `__index` metamethod in `domObject_push_metatable()` retrieved the value correctly via `Object::get(key)`, then called `lua_replace(L, 1)` to move the result into the userdata's slot. `lua_replace` also pops the top, so, on return, the key string was at the top of the stack and Lua picked it up as the metamethod's single return value, making every field access on a `dom::Object` userdata silently return the key it was asked for. This was latent until now because no Lua script in the test suite previously read fields off a `dom::Object` userdata. Surfaced while wiring corpus extensions: a script doing `corpus.symbols[i]` saw `"symbols"` (the key) instead of the array.
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
2 times, most recently
from
ad08eed to
db26b0d
Compare
This adds a hook that runs user-provided Lua scripts after corpus extraction and finalization, before any generator runs. Extensions live in <addon>/extensions/*.lua for each addon root in the configuration. A script may define `transform_corpus(corpus)`, which is invoked once with a flat DOM view of the corpus. The script may mutate the corpus by calling pre-registered globals on the `mrdocs` table; currently: - `mrdocs.set_brief(symbol_id, text)`: replace a symbol's brief with a single-paragraph plain-text block. Each setter validates its arguments and raises a Lua error on misuse; any uncaught error in a script aborts the build. Multiple extensions run in alphabetical order by file path. The mutation surface is intentionally narrow; additional setters will land as concrete use cases surface. A golden fixture (test-files/golden-tests/extensions/lua-set-brief/) rewrites a function's brief from Lua and verifies the change reaches the xml output. Finally, this touches Lua wrapper for two new affordances: `Scope::pushDom` for the corpus argument, and a `Context::nativeState()` escape hatch for binding native C functions as Lua globals (the wrapper doesn't abstract that yet).
This mirrors the Lua corpus-mutation hook for JavaScript. Extension scripts under <addon>/extensions/*.js can now define `transform_corpus(corpus)` and call `mrdocs.set_brief(symbol_id, text)`, just like their Lua counterparts. The Lua and JS bindings now share a language-agnostic `setBriefImpl` helper that takes already-extracted `dom::Value` arguments. Each binding is a thin adapter: - Lua: the existing C closure registered on the raw `lua_State*`. - JS: a `dom::Function` exposed as a property of a `mrdocs` global object; the wrapper's `setGlobal` -> `toJsValue` -> `makeFunctionProxy` chain handles the rest. Discovery picks up both *.lua and *.js, sorted together so script ordering doesn't depend on the chosen language. A golden fixture mirrors the Lua test.
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
3 times, most recently
from
e58fb63 to
075f7bc
Compare
There was a problem hiding this comment.
The PR is great. It's really nice to see we already have examples that transform the corpus via the scripts.
I don't see much value in putting Lua scripts as Handlebars helpers (a cheap improvement win in an extension type that already exists) with Corpus-mutation extensions in Lua and JavaScript (a completely new feature that's huge).
a new
Describe.hpp
Doesn't it already exist? And wasn't the distinction between Describe.hpp and other headers related to reflection that Describe.hpp would only be about porting Boost.Describe?
MRDOCS_DESCRIBE_HIERARCHY
I believe this needs better in-source documentation. We even support that now (or are about to).
I also don't understand these *Hierarchy.hpp files that much. The PR describes that they exist, but not why they exist. All I know is they're using this pattern of including everything, which defeats the purpose of public headers. I also don't see why the user ever needs that information to be public.
They also seem to defeat the purpose of things in a sense. Because we're trying to avoid manually listing things with reflection, but we are still doing it there. The typical pattern we use for that is a single source of truth as inc files.
I also don't understand how a hierarchy can be described as a list unless there's some special convention about this list.
Tests
There's enough new functionality (even public functionality) here that seems to deserve unit tests.
Golden tests
Do they really cover everything that's possible? What are the typical use cases for this? Do we get errors on invalid transformations? How do we merge symbols in the corpus? How do you add symbols to the corpus?
Third-party:
third-party/patches/lua/CMakeLists.txtupdated for the vendored Lua build.
What do you mean by vendored Lua build here?
- No CI workflow changes needed — the existing golden-test job runs all of
test-files/golden-tests/on every build, so the new trees stay covered automatically going forward.
Documentation
I think the documentation should make the same distinction you made here. It could maybe be described as corpus-mutation extensions or something. Because we have this concept of extensions. Then, things people understand as extensions are these customizations of the template system and the corpus-mutation extensions. They are very different layers and share a common logic of knowing where to put addons, etc. They also have to know about supplemental addons, which they're probably going to use the most. The documentation concerns the user journey.
Also, any reference should be generated automatically. I don’t understand the API very well. Is mrdocs.set the only function we have?
I also don’t see the pattern we had discussed about taking inspiration from Lua/Darktable. It seems the new extension system uses the same pattern as the helpers extension system, but that pattern is not appropriate there. What if an extension wants to provide different functionalities? At a minimum, it's important to include a comparison and an explanation of why the PR chose a different strategy.
Is that the best Python and Lua API we could have on the Lua/JS side? What are the alternatives? They look very different from what things look like on the C++ side. Are they extendable, or will we need to break the Lua/JS API every time there’s a new feature?
The "Corpus argument" section describes a subset of a symbol's API (I don't really understand why, since the full reference is elsewhere), but the API of the corpus object itself isn't described.
|
Thanks for the thorough review. Great stuff in there :-). Here's a point by point reply:
Open decisions for you:
|
This completes the work on extensions by making both the read and the write view of scripts lay on the describe machinery already used by the rest of the code. Scripts can now read any described field, and write any field on an (intentionally small) allowlist.
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
4ae57c0 to
22b6462
Compare
This adds a MRDOCS_DESCRIBE_HIERARCHY macro that registers the derived classes of a polymorphic base. This lets generic code dispatch over the closed set of derived types without the per-base X-macro boilerplate every consumer would otherwise need.
This wires the macro added in the previous commit to every polymorphic base in MrDocs. Each registration lives in a small private include under src/lib/Metadata/. The hierarchy info is a compile-time consumer-side concern, so no public header exposes it; consumers include the relevant file directly. This commit only registers the relationships. The next one will introduce the first consumer.
The previous step's generic setter could navigate through `Optional`, `vector` and described structs, but not `Polymorphic<T>`, the type used throughout MrDocs for type-erased value pointers. It can now. The generic setter dispatches a `kind:` field against the closed set of derived classed registered via `MRDOCS_DESCRIBE_HIERARCHY`, forwards the remaining DOM keys to the matched class, and writes the result into the `Polymorphic<T>`. The Lua adapter gains a recursive table-to-DOM converter; `doc` and `loc` are added to the allowlist.
Some library types are better presented in documentation by their
semantic role than by their literal C++ form: e.g., a function returning
`capy::task<T>` is more useful to readers when documented as a coroutine
yielding `T` than when shown with its literal task-object return type.
Hence, this adds `returnType` to the script-side allowlist.
This is the substrate for semantic-rendering extensions, not the fully
declarative API one might eventually expose ("declare this type an
awaitable, MrDocs handles the rest"). That higher-level API can be built
on top of this hook.
Replace the duplicated subset of symbol-fields with a clear "what `corpus` actually is" lead. Note explicitly that lookups, name searches and other queries are not exposed on `corpus`; scripts walk `corpus.symbols` and filter. For per-symbol fields, link out to the templates documentation rather than duplicate a subset.
Template helpers (generators.adoc) and corpus-mutation extensions (extensions.adoc) live at different layers but share the same addon-discovery machinery. Both pages were duplicating the explanation of `addons` and `addons-supplemental`, and neither covered the user journey of "what is an addon, where does it sit?". This adds `addons.adoc` as the canonical page for the addon concept: the built-in addon, replacement via `addons`, supplementation via `addons-supplemental`, the on-disk layout, and the discovery rule (last-wins for templates/helpers, full aggregation for extensions). `extensions.adoc` and `generators.adoc` drop their local explanations and cross-link to it.
gennaroprota
force-pushed
the
feat/support_scripting_extensions
branch
from
22b6462 to
f68a074
Compare
Up to you. I'll let you come up with the best design you can and we can iterate on it. |
3 participants
This mirrors the existing JavaScript helpers for Lua, and adds support for corpus-mutation extensions written either in Lua or Javascript.
Adds two related script-extension capabilities and the supporting infrastructure to expose mrdocs' reflection model to those scripts:
addons/extensions/run after the corpus is built and can rename, retag, or otherwise mutate symbols before generation.To make the script surface coherent rather than a hand-maintained binding table, the extension layer is driven by reflection: a new
Describe.hppexposes types through their reflection metadata, a newMRDOCS_DESCRIBE_HIERARCHYmacro lets polymorphic bases describe their derived hierarchy, and the existingPolymorphic<T>type is now usable frommrdocs.set. Scripts therefore see a typed view of the model that stays in sync with the C++ types automatically as new symbol/type kinds are added.A small
dom::Objectbug surfaced during the work — field lookup was returning the key instead of the value — and is fixed in the same PR.Changes
src/lib/Extensions/(RunExtensions.{cpp,hpp}) runs Lua/JS corpus-mutation scripts;CorpusImpl.cppinvokes it after corpus construction.src/lib/Support/Lua.cppandinclude/mrdocs/Support/Lua.hppgrow to support binding mrdocs' reflection types into Lua (includingPolymorphic<T>).src/lib/Gen/hbs/Builder.cppupdated to load Lua helpers alongside the existing JS ones. Newinclude/mrdocs/Support/Describe.hpp, theMRDOCS_DESCRIBE_HIERARCHYmacro, and seven new*Hierarchy.hppheaders underinclude/mrdocs/Metadata/apply the macro to symbols, types, names, template args/params, and doc-comment blocks/inlines.dom::Objectfield-lookup fix.extensions/js-set-name/andextensions/lua-set-name/exercise corpus-rename scripts in JS and Lua. Newgenerator/hbs/lua-helper/andgenerator/hbs/lua-helper-layering/exercise Lua Handlebars helpers, including helper layering across multiple addon directories. All intentional output additions, not regenerations.docs/modules/ROOT/pages/extensions.adocdocumenting the scripting model;generators.adocupdated for the Lua helper surface and the layering model.third-party/patches/lua/CMakeLists.txtupdated for the vendored Lua build.Describe.hpp, the*Hierarchy.hppset) which is purely additive. Downstream code that specialises reflection traits manually should check whetherMRDOCS_DESCRIBE_HIERARCHYnow applies to types it was previously describing by hand.Testing
test-files/golden-tests/extensions/andtest-files/golden-tests/generator/hbs/lua-helper*/are the primary coverage: they execute real Lua and JavaScript scripts against fixed input fixtures and assert the resulting.xml/.html/.adocoutput byte-for-byte. Any future regression in helper or extension behavior fails these tests.test-files/golden-tests/on every build, so the new trees stay covered automatically going forward.Documentation
docs/modules/ROOT/pages/extensions.adocdocuments corpus-mutation extensions in both Lua and JavaScript, the lookup paths (addons/extensions/), and the available script API.docs/modules/ROOT/pages/generators.adocupdated to cover the Lua helper surface alongside the existing JS one, and the helper-layering model across multiple addon directories.