Vendor pre-generated ODS output in the melior fork and solx-mlir — get tblgen out of proc-macro expansion for good

[nebasuke]

Summary

melior-macro's dialect! macro runs TableGen inside rustc's process: the proc-macro crate links LLVM's libTableGen (C++) via tblgen-rs, parses .td files at macro-expansion time, and emits the typed op wrappers. A proc-macro that links LLVM C++ is the architectural oddity behind an entire class of Windows CI breakage, of which we have now patched three symptoms individually:

Symptom	Failure point	Workaround currently in place
#550	link time — GCC 16 COMDAT collision linking the proc-macro cdylib against static libstdc++	rm libstdc++.a in build-toolchain/action.yml (forces shared libstdc++)
#589	load time — rustc's stale bundled libwinpthread-1.dll shadows MSYS2's runtime when LoadLibraryExW resolves the proc-macro's DLL chain	rm libwinpthread-1.dll (#590)
NomicFoundation/solx-llvm#135	link time — no-assertions Release LLVM compiles out Record/RecordVal/Init::dump(), which tblgen-rs references; PE linking resolves eagerly	keep dump() in no-assertions builds (LLVM fork patch, verified by #619)

Each fix is correct, but the class remains: the proc-macro is one toolchain-drift or cache-fingerprint change away from the next symptom. The structural fix flagged in #589's follow-ups is to remove tblgen from macro expansion entirely.

The output of dialect! is a pure function of pinned inputs — the .td files (LLVM pinned at 21.1; SolOps.td/YulOps.td pinned by the solx-llvm submodule) and the generator code (melior fork pinned by rev). Nothing about it needs to run on the consumer's machine, let alone inside rustc. So: run the generation offline, commit the pretty-printed Rust, and delete the macro invocations.

Current state

melior-macro exports eight proc-macros. Seven are pure Rust token generation. Only dialect! uses tblgen (macro/src/dialect.rs): it builds a TableGenParser, parses the named .td files out of LLVM_INCLUDE_DIRECTORY (baked in by the macro crate's build.rs via llvm-config --includedir), walks all Op-derived records, and emits a module of op wrappers, type-state builders, and a per-dialect operation enum. Because Cargo links a crate's dependencies into the proc-macro cdylib, this one macro is why melior_macro.dll carries LLVM C++.

Two invocation sites:

• melior fork — melior/src/dialect/ods.rs: ~40 dialect! invocations for the upstream MLIR dialects, behind the ods-dialects feature.
• solx-mlir — src/ods.rs: two invocations for the sol and yul dialects, with .td files from solx-llvm's MLIR tree.

Plan

1. Split melior-macro into a melior-ods-codegen library crate (keeps the tblgen dep; the entry point generate_dialect(DialectInput) -> TokenStream is already cleanly factored) and the melior-macro proc-macro crate, which keeps only the seven pure-Rust macros. melior-macro loses its tblgen dependency and its build.rs entirely — no native links, no LLVM needed to build it.
2. Add a standalone generator binary (xtask-style, e.g. melior-ods-gen) depending on melior-ods-codegen: takes the same {name, files, include_directories} tuples, runs the existing code path as a normal process, pretty-prints with prettyplease, writes .rs files.
3. Commit the generated output at both sites:
   • melior fork: replace the ods.rs invocations with committed generated modules. Consider trimming to the dialects solx-mlir actually uses — the full set is a lot of vendored code for dialects we will never touch.
   • solx-mlir: replace src/ods.rs with committed ods/sol.rs / ods/yul.rs. Natural home for the regen command is solx-dev, since the TABLEGEN_210_PREFIX/MLIR_SYS_210_PREFIX plumbing already exists in .cargo/config.toml.
4. CI staleness check: regenerate and diff against the committed copy (same pattern as EDR's "typings file is up to date" job). Load-bearing, not optional — see cadence note below.
5. Mark generated files linguist-generated in .gitattributes.

What this makes deletable

• The Windows: freshly-built melior-macro proc-macro fails to load (E0463) — rustc's stale bundled libwinpthread-1.dll shadows MSYS2's GCC-16 runtime #589/ci(windows): drop rustc's stale libwinpthread so C++ proc-macros load #590 rm libwinpthread-1.dll hack — no C++ in the proc-macro means no DLL chain for rustc to fail loading.
• The solx-llvm#135 keep-dump() patch — nothing in any consumer build links tblgen anymore; only melior-ods-gen does, at regen time, on a dev box, where an assertions build is fine.
• Possibly the Windows: fresh llvm-sys builds cannot link against the bundled static libstdc++.a under MSYS2 GCC 16 — CI is one cache-fingerprint change from breaking #550 rm libstdc++.a workaround — if the proc-macro cdylib was the only link site hitting the COMDAT collision. Needs re-verification against the Windows: fresh llvm-sys builds cannot link against the bundled static libstdc++.a under MSYS2 GCC 16 — CI is one cache-fingerprint change from breaking #550 write-up before removal; test executables still link MLIR's C API statically via mlir-sys, which is a different link than the proc-macro was.
• The requirement that llvm-config and the full MLIR .td include tree be present just to compile melior. Clean builds get faster too: no TableGen parsing inside rustc.

The failure class — "toolchain drift under the cache breaks the proc-macro" — dies on all platforms, instead of being patched per-symptom on Windows.

Caveats

• Sol/Yul regen is not rare. The "LLVM is pinned, regeneration is rare" argument holds for melior's upstream MLIR dialects, but SolOps.td/YulOps.td are under active development — every new op means a regen commit riding along in the PR. Hence the CI check; same living arrangement as EDR's committed index.d.ts, including the same failure mode it catches (edit the .td, forget to regen).
• Bulk. The full ODS set pretty-printed is large; trimming to used dialects helps materially.
• Determinism. tblgen record order is deterministic given identical inputs and prettyplease is stable, but the generator must run against the same LLVM pin the .td files come from.
• Fork divergence. The upstreamable variant (move tblgen to build.rs + include!; raviqqe/melior could take it) only fixes the load-time class — build scripts are separate processes, but they still link tblgen, so the solx-llvm#135 patch would have to stay. Since we already maintain the fork and pin LLVM, vendoring is the variant that pays for itself.

References

• Windows: freshly-built melior-macro proc-macro fails to load (E0463) — rustc's stale bundled libwinpthread-1.dll shadows MSYS2's GCC-16 runtime #589 (root-cause analysis; this issue is its "structural follow-ups" item)
• ci(windows): drop rustc's stale libwinpthread so C++ proc-macros load #590 (tactical workaround this deletes)
• Windows: fresh llvm-sys builds cannot link against the bundled static libstdc++.a under MSYS2 GCC 16 — CI is one cache-fingerprint change from breaking #550 (parent link-time issue)
• [TableGen] Keep the dump() methods in no-assertions builds solx-llvm#135 (tblgen dump() link fix this makes unnecessary)
• ci: fold benchmark mirror into the build-llvm composite action #619 (end-to-end verification of the current workaround stack)