Windows: fresh llvm-sys builds cannot link against the bundled static libstdc++.a under MSYS2 GCC 16 — CI is one cache-fingerprint change from breaking

[nebasuke]

Summary

Any Windows build that actually recompiles llvm-sys fails to link solx with:

ld.exe: target-llvm/target-final/lib/libstdc++.a(string-inst.o): multiple definition of
  `std::__cxx11::basic_string<…>::_M_replace_cold(…)';
  libllvm_sys-*.rlib(ExecutionEngine.cpp.obj): …/mingw64/include/c++/16.1.0/bits/basic_string.tcc:545: first defined here
collect2.exe: error: ld returned 1 exit status

This affects dev and release profiles alike. test.yaml's Windows leg is currently green only because its Swatinem cargo cache keeps llvm-sys "Fresh" — the moment anything dirties llvm-sys's cargo fingerprint (profile/env change, RUSTFLAGS, an llvm-sys or inkwell bump, a cache eviction), Windows CI breaks with no code change to blame. We hit this while building a cross-OS solx-tester workflow (cross-OS plan, Stage 1).

Root cause

build-llvm/action.yml copies MSYS2's libstdc++.a into the LLVM install tree (LIBSTDCPP_SOURCE_PATH → target-llvm/target-final/lib/libstdc++.a, handled by solx-dev/src/llvm/platforms/x86_64_windows_gnu.rs) so release binaries can link C++ statically and be self-contained. Because llvm-sys's build script emits -L target-llvm/target-final/lib, that static archive shadows MSYS2's own -lstdc++ resolution (which would prefer the shared libstdc++.dll.a) for every crate link in the workspace.

Under MSYS2's current GCC 16, this is fatal for freshly compiled C++ objects: GCC 16 added basic_string::_M_replace_cold and emits it as a COMDAT in user objects, which collides with the strong definition in the static archive's explicit-instantiation objects (string-inst.o / wstring-inst.o) whenever those members get pulled into the link.

It is not version drift. We ran the controlled experiment:

Bundled libstdc++.a	llvm-sys objects	Result
from cache (older GCC)	fresh, GCC 16.1.0, release	❌ multiple definition (run)
from cache (older GCC)	fresh, GCC 16.1.0, dev	❌ identical (run)
refreshed to the runner's own GCC 16.1.0 copy	fresh, GCC 16.1.0	❌ identical (run)
removed (link shared libstdc++.dll.a)	fresh, GCC 16.1.0	✅ builds; solx-tester passes 74,605 cases (run)

Same-version static fails while shared succeeds → the conflict is structural (static explicit-instantiation archive vs GCC-16 COMDAT-emitting objects on MinGW), not a stale cache. A cache built pre-GCC-16 has the same outcome for the same reason.

Reproducing

On windows-2025 with the repo's standard build steps (prepare-msys, cached LLVM/solc), force llvm-sys to recompile — any fingerprint change does it, e.g. CARGO_PROFILE_RELEASE_STRIP=none cargo build --release --target x86_64-pc-windows-gnu --bin solx. Link fails as above.

Fix options

1. Stop bundling libstdc++.a into the cached LLVM tree. Keep builds linking the shared libstdc++ (works today, proven above); do the static bundling only where the self-contained property is needed — in the release-packaging workflow, which builds LLVM in-run and is therefore always ABI-consistent with itself. Least ongoing maintenance.
2. Keep the bundle but make CI robust: delete/ignore the bundled .a in test builds (rm after cache restore — our current workaround), accepting that test binaries link shared while release binaries link static.
3. Keep the bundle and try to restore same-version linking (e.g. investigate why the COMDAT/explicit-instantiation conflict is fatal here — linker flags, -fno-implicit-templates, or GCC 16 behavior); highest effort, and our experiment suggests same-version isn't sufficient as-is.

Related history: the MSYS2 pinning in prepare-msys (#374, #530) and the msys-hash cache-key component protect against action-file changes, but pacman installs live package versions, so the toolchain underneath the cache key still drifts — worth folding gcc --version into the Windows cache key regardless of the option chosen.

Found while executing the cross-OS tester plan; the Windows spike branch (win-tester-spike) contains the working configuration.