Skip to content

test(edr_provider): broaden solx provider-path stack-trace coverage - #1552

Open
nebasuke wants to merge 4 commits into
mainfrom
feat/solx-stack-trace-provider-coverage
Open

test(edr_provider): broaden solx provider-path stack-trace coverage#1552
nebasuke wants to merge 4 commits into
mainfrom
feat/solx-stack-trace-provider-coverage

Conversation

@nebasuke

@nebasuke nebasuke commented Jul 10, 2026

Copy link
Copy Markdown
Member

Adding wide test coverage for stack trace tests, along with some test refactoring for more re-use. This is a follow-up of #1572. I've tried to put tests into blocks to make it a bit easier to follow what's currently covered. I could split into files instead if that is easier.

Note that I've removed two tests as they were already covered by a more generic test, see also the test folding Claude section below.

This is the first PR of a planned few. I'm trying to do a structured expansion of stack-trace coverage. The idea here is that this functions both as increased coverage for solx/solc, and some of the pre-work for being able to migrate away from hardhat-tests.

Planned next, roughly in order:

  1. Failing deployments: sending ether to a non-payable constructor, or deploying with wrong or truncated constructor arguments. The error is attributed by looking at where in the bytecode execution stopped, so solx's debug info can place it differently than solc's. This is currently untested.
  2. Solc-specific pattern checks running on solx bytecode: a few error classifications work by recognizing short, hard-coded instruction sequences that solc happens to emit (for example "you called an address with no contract on it"). solx compiles the same Solidity to different instructions, but these checks still run on its traces. Nothing today verifies whether they match correctly, match wrongly, or simply never fire. Includes the plain "call an address with no contract behind it" case itself.
  3. Two rarer error kinds: a revert with a custom error the decoder doesn't recognize, and panic 0x51 (calling a function variable that was never assigned). Both are reported through fallback code paths no test reaches today.
  4. Errors travelling between contracts: a nested call failing because it ran out of gas, errors bubbling up through proxy contracts, calls that fail before the callee even runs, and deployments rejected for exceeding the contract size limit.
  5. eth_call: the tests so far only cover failing transactions; failing read-only calls take a separate path that we should either cover or explicitly leave out of scope.

Claude summary

Rebased onto main now that #1577 (the DWARF attribution fixes + fixture infrastructure) is merged. This PR is tests only: it broadens the provider-path (JSON-RPC) stack-trace coverage for solx artifacts from 11 tests to 36, all against fixtures already on main — and organizes the file so the coverage is auditable against the inference pipeline instead of reading as a blob.

The provider path is what Hardhat 3's JS-test flow (hardhat test + ethers/viem) hits. #1425 covers the solidity-test-runner path via the JS parity sweep, but the sweep is CI-skipped until @nomicfoundation/hardhat-solx is published — so these are the trace assertions that actually run in CI today.

File layout mirrors the inference pipeline

solx_stack_trace.rs puts all helpers in one block up front, then one test section per edr_solidity inference stage, so reviewing the file is walking the pipeline:

Section Inference path pinned Tests
Provider plumbing end-to-end smoke on the minimal Counter fixture 1
Pre-execution guards infer_before_tracing_call_message: payability, missing function/fallback/receive, direct library calls 6
Calldata decoding check_last_instruction: InvalidParamsError for truncated calldata 1
Revert/panic/custom check_revert_or_invalid_opcode: six panic codes with statement anchors, custom-error argument decoding, revert-line discrimination between requires 8
Callstack reconstruction frame push/pop, submessage splicing, filter_redundant_frames: cross-contract frames, external/internal/mutual recursion, internal helpers and libraries, linked external library (linkReferences + DELEGATECALL decode), fallback/receive bodies 9
Modifiers fix_initial_modifier + SolxTraceStrategy attribution: plain, nested, cross-contract, bare-revert 4
Create path creation-code debugInfo: reverting constructors, plain and via internal helper 2
Submessage checks check_last_submessage: returndata-size errors (real callee and EOA target) 2
-O3 twins (the mode3_* tests) three modifier scenarios recompiled at solx optimizer mode 3 (-O3; hardhat-solx defaults to -O1). Since solx 0.1.6 the -O1 DWARF maps compiler helpers to real statement lines, so only these more aggressively optimized artifacts still reach the fallback inference paths: walking back to the last statement when a revert is attributed to a declaration line, and recovering the failing function for unmapped bare reverts 3

Every test now asserts a source anchor (function/contract/statement line), not just the entry variant — e.g. each panic code pins the statement it's attributed to, and the dispatch guards pin the declaration they anchor at. The two call shapes where frame gains/losses are the point (cross-contract call, external recursion) pin the full rendered trace via assert_trace_shape. Line pins are goldens: a shifted anchor after a solx upgrade is expected drift (update the pin), a lost frame is a regression.

Deliberately not covered — location-free classification paths already pinned by the solc corpus, plus follow-up material: message-kind dispatch (precompiles, unrecognized contracts, contract-too-large), create-side guards, the solc-opcode-pattern rewrites in mapped_inline_internal_functions_heuristics, out-of-gas rewrites, proxy propagation, and the eth_call channel. bail_on_call_failure was dropped from the test provider config since nothing exercised it.

The external-recursion test already caught one real regression: a late refactor in #1425 dropped the per-strategy recursion carve-out in filter_redundant_frames, collapsing any external-recursion depth to a single frame. That fix was upstreamed separately as #1569.

Two tests folded, one renamed (no coverage removed)

The diff deletes two tests by name; both were variant-only assertions strictly subsumed by a stronger replacement:

  • panic_error_variant_surfaces_for_overflow_scenario asserted only that some PanicError entry appears for OverflowTest — a weaker duplicate of the panic-code family right next to it. It's now panic_code_surfaces_for_arithmetic_overflow in that family, additionally pinning the code (0x11) and the statement the panic is attributed to (x = x + 1, line 26).
  • custom_error_variant_surfaces_for_custom_error_scenario asserted only that some CustomError entry appears. It's now custom_error_decodes_name_and_args, additionally pinning the decoded message (reverted with custom error 'MyError(42, "custom error")') and the revert statement line — the known-selector argument-decoding path had no coverage before.
  • The third test from the same old "variance-axis" block, revert_error_variant_surfaces_for_counter, is kept as revert_error_surfaces_end_to_end_for_counter: it's the only test on the minimal Counter fixture (a second, independent artifact assembly), so it stays as the plumbing smoke test.

Accepted gaps

Two SolxTraceStrategy fallbacks remain untested: unresolved_callstack_entry and the panic-helper selector fallback. Both are last resorts for instructions whose DWARF location doesn't resolve to a known function, and no fixture produces that — not even at -O3 (MutualRecursionTest, the closest candidate, reached parity without them). Auditing that solx covers every reachable PC with a DW_TAG_subprogram belongs in solx's own test suite, not EDR scenarios.

Verification

@changeset-bot

changeset-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 6ee957d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 10, 2026 14:20 — with GitHub Actions Inactive
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.89%. Comparing base (bdcf6fb) to head (6ee957d).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1552      +/-   ##
==========================================
+ Coverage   79.77%   79.89%   +0.11%     
==========================================
  Files         453      452       -1     
  Lines       79038    78956      -82     
  Branches    79038    78956      -82     
==========================================
+ Hits        63050    63079      +29     
+ Misses      13810    13694     -116     
- Partials     2178     2183       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nebasuke nebasuke added the no changeset needed This PR doesn't require a changeset label Jul 10, 2026
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 10, 2026 14:45 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 10, 2026 14:45 — with GitHub Actions Inactive
Base automatically changed from feat/solx-dwarf-debug-info to main July 20, 2026 12:15
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from e7c64db to dc3c75e Compare July 20, 2026 14:47
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 20, 2026 14:47 — with GitHub Actions Inactive
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 20, 2026 15:14 — with GitHub Actions Error
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 20, 2026 15:14 — with GitHub Actions Error
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 20, 2026 15:47 — with GitHub Actions Inactive
@nebasuke nebasuke changed the title test(edr_provider): broaden solx provider-path stack-trace coverage test(edr_provider): broaden solx provider-path stack-trace coverage; fix declaration-attributed inference Jul 20, 2026
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 20, 2026 16:16 — with GitHub Actions Error
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 20, 2026 16:16 — with GitHub Actions Error
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from e09dae7 to ab5a7d7 Compare July 20, 2026 17:40
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 20, 2026 17:40 — with GitHub Actions Inactive
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 20, 2026 17:47 — with GitHub Actions Error
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 20, 2026 17:47 — with GitHub Actions Error
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from ab5a7d7 to 90deafb Compare July 20, 2026 18:09
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 20, 2026 18:09 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 20, 2026 18:42 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 20, 2026 18:42 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 22, 2026 17:42 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 22, 2026 17:44 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 22, 2026 17:44 — with GitHub Actions Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Improves solx stack-trace attribution/parity (especially around compiler-generated “declaration-attributed” locations) and adds tooling/fixtures to regenerate and test these behaviors against updated solx versions.

Changes:

  • Extend solx trace strategy + DWARF decoding to better handle declaration-padding / unmapped locations and line-0 call sites.
  • Add a CLI command to regenerate solx compiler-output fixtures, plus a new StackTraceScenarios fixture corpus.
  • Expand provider integration tests and update sweep/docs to reflect improved parity and solx 0.1.6 behavior.

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
js/integration-tests/solx-parity-sweep/test/sweep.ts Removes scenarios that no longer diverge from solc, tightening parity checks.
js/integration-tests/solx-parity-sweep/README.md Updates pinned-divergence documentation to reflect solx 0.1.6 parity changes.
crates/tool/cli/src/update.rs Fixes repo-root discovery by adjusting ancestor depth.
crates/tool/cli/src/solx_fixtures.rs Adds fixture-regeneration helper that runs solx --standard-json and rewrites outputs.
crates/tool/cli/src/main.rs Exposes fixture regeneration as a CLI subcommand.
crates/edr_solidity/src/trace_strategy.rs Adds strategy hooks for location equivalence and declaration-attributed inference; improves revert attribution.
crates/edr_solidity/src/error_inferrer.rs Uses new strategy hooks to keep stack traces stable and avoid declaration-location regressions.
crates/edr_solidity/src/debug_info/dwarf.rs Handles solx 0.1.6 call_line=0 by falling back to decl sites; adds targeted tests.
crates/edr_solidity/fixtures/sources/StackTraceScenarios.sol Adds new Solidity scenario corpus targeting provider-path stack-trace variants.
crates/edr_solidity/fixtures/solx_compiler_output_stack_trace_scenarios.json Adds compiled solx output fixture for StackTraceScenarios.
crates/edr_solidity/fixtures/solx_compiler_output.json Updates Counter fixture output to new solx/toolchain settings.
crates/edr_solidity/fixtures/solx_compiler_input_stack_trace_scenarios.json Adds committed standard-json input for StackTraceScenarios (with contents scrubbed).
crates/edr_solidity/fixtures/solx_compiler_input.json Pins optimizer mode in Counter standard-json input for fixture stability.
crates/edr_solidity/fixtures/README.md Documents new gen-solx-fixtures regeneration path and the new fixture pair.
crates/edr_provider/tests/integration/solx_stack_trace.rs Broadens provider-level integration tests for more stack-trace variants and new fixture.
.changeset/solx-declaration-attributed-inference.md Adds changelog entry for the solx stack-trace inference fixes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/tool/cli/src/update.rs
Comment thread crates/tool/cli/src/solx_fixtures.rs
Comment thread crates/tool/cli/src/solx_fixtures.rs
Comment thread crates/tool/cli/src/solx_fixtures.rs
@nebasuke
nebasuke marked this pull request as ready for review July 24, 2026 18:49
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from 7dde91b to 39b9358 Compare July 25, 2026 13:30
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 13:30 — with GitHub Actions Inactive
@nebasuke
nebasuke changed the base branch from main to fix/solx-dwarf-attribution July 25, 2026 13:30
@nebasuke
nebasuke marked this pull request as draft July 25, 2026 13:30
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 25, 2026 13:32 — with GitHub Actions Error
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 13:32 — with GitHub Actions Inactive
@nebasuke
nebasuke force-pushed the fix/solx-dwarf-attribution branch from a4d67d2 to b1de264 Compare July 25, 2026 13:57
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from 39b9358 to 33c8322 Compare July 25, 2026 13:57
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 13:58 — with GitHub Actions Inactive
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 25, 2026 14:02 — with GitHub Actions Error
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 25, 2026 14:02 — with GitHub Actions Error
@nebasuke
nebasuke force-pushed the fix/solx-dwarf-attribution branch from b1de264 to a0d6af5 Compare July 25, 2026 14:26
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from 33c8322 to 0d2f30a Compare July 25, 2026 14:26
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 14:27 — with GitHub Actions Inactive
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 25, 2026 14:31 — with GitHub Actions Failure
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 25, 2026 14:31 — with GitHub Actions Error
@nebasuke
nebasuke force-pushed the fix/solx-dwarf-attribution branch from a0d6af5 to fb576c7 Compare July 25, 2026 15:21
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from 0d2f30a to 052a2b6 Compare July 25, 2026 15:21
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 15:21 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 15:25 — with GitHub Actions Inactive
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 25, 2026 15:25 — with GitHub Actions Inactive
@nebasuke
nebasuke force-pushed the feat/solx-stack-trace-provider-coverage branch from 052a2b6 to b8faf71 Compare July 25, 2026 17:10
nebasuke added 4 commits July 28, 2026 12:03
Groundwork for the coverage in this PR, with no change in what the
existing tests assert:

- The three fixture loaders differed only in paths and source key; they
  now delegate to `assemble_build_info`. The `include_str!` literals stay
  at the call sites — the macro needs a literal path.
- Drop the local `deploy` in favour of `test_utils::deploy_contract`,
  which does the same send-and-read-receipt dance.
- Drop the local `source_reference_of` match in favour of the existing
  `StackTraceEntry::source_reference` method.
- Extract `stack_trace_from_failure` from `expect_failed_call_stack_trace`
  and route the latter through a value-taking variant, so the
  deployment-failure and payability helpers added later can reuse both.
- Calldata construction moves into `call` / `encode_call_u256` /
  `encode_call_address` instead of being spelled out per test.
One test per DWARF-decoding axis over the existing scenarios fixture (no
fixture changes): panic sub-codes (0x01/0x12/0x21/0x31/0x32), CREATE-path
reverts (plain and internal-helper constructors — creation-bytecode
`debugInfo` was previously untested), modifier bodies, cross-contract and
mutually recursive CALL chains, external-recursion frame counts, internal
helpers/recursion, internal libraries, fallback/receive bodies, and
per-statement line discrimination (two `require`s in one function; the
trace must point at the second).

The provider path is what Hardhat 3's JS-test flow (`hardhat test` +
ethers/viem) hits. #1425 covers the solidity-test-runner path via the JS
parity sweep, but the sweep is CI-skipped until
`@nomicfoundation/hardhat-solx` is published — so these are the trace
assertions that actually run in CI today.

Assertions stay at entry-variant + source line, except for the two call
shapes where frame gains/losses are the point (cross-contract call,
external recursion): those pin the full rendered shape via
`assert_trace_shape`. Exhaustive frame parity with solc remains the
sweep's job.
… artifacts

The StackTraceScenarios fixture's dispatch-level contracts, so far unused:
the entry variants the solc corpus exercises but no solx test reached.
FunctionNotPayableError, UnrecognizedFunctionWithoutFallbackError,
MissingFallbackOrReceiveError, both FallbackNotPayable variants,
InvalidParamsError (truncated calldata), plus an externally linked library
exercising `linkReferences` placeholder substitution and DELEGATECALL
decoding into the library's own `debugInfo`, and DirectLibraryCallError
for a direct call to a deployed library.

`link_library` bails on bytecode referencing more than one distinct
library instead of silently mis-linking — it takes a single address.
…erence pipeline

Organize the file so coverage is auditable against edr_solidity's
inference stages: all helpers in one block up front, then one test
section per stage, with a file-level coverage map that also states
what is deliberately left to the solc corpus or follow-ups.

Fold the two variant-only variance-axis tests into their stronger
families (overflow becomes panic code 0x11; the custom-error test now
pins the decoded message and statement line). Strengthen the panic
family and dispatch guards to pin source anchors, not just entry
variants. Drop the unused bail_on_call_failure from the test provider
config — the eth_call channel is follow-up material.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

crates/edr_provider/tests/integration/solx_stack_trace.rs:74

  • assemble_build_info uses unwrap() when replacing the source content. If the fixture format changes or the source_key is wrong, this will panic without a helpful error message, making failures harder to diagnose in CI. Prefer returning a contextual anyhow error instead.
    input.sources.get_mut(source_key).unwrap().content = source_content.to_owned();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no changeset needed This PR doesn't require a changeset

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants