Skip to content

feat: add backwards compatibility for Response::traces - #1301

Open
Wodann wants to merge 10 commits into
mainfrom
feat/tracing-back-compat
Open

feat: add backwards compatibility for Response::traces#1301
Wodann wants to merge 10 commits into
mainfrom
feat/tracing-back-compat

Conversation

@Wodann

@Wodann Wodann commented Feb 19, 2026

Copy link
Copy Markdown
Member

Resolves #1288.

This reintroduces the Response::traces getter function to maintain support for Hardhat 2's EthereumJS VM interface.

I tested this using the help of Claude:

Hardhat Plugin Compatibility Report — EDR Tracing Back-Compat

Date: 2026-02-20
EDR branch: feat/tracing-back-compat
Hardhat branch: hh2/tracing-unification

Overview

This report validates that the new EDR trace format (unified tracing architecture)
is backward-compatible with the existing Hardhat v2 plugin ecosystem. Testing was
performed at three levels:

  1. Plugin test suites — each plugin's own tests run against local Hardhat+EDR
  2. Third-party project tests — real-world projects using stock plugins
  3. Third-party projects with linked plugins — full local stack (EDR + Hardhat + plugins)

Critical Finding: includeCallTraces Default

During testing, solidity-coverage's test suite revealed that Response::traces()
returned empty arrays because include_call_traces defaults to IncludeTraces::None
in the new architecture. This was fixed in Hardhat commit c46cf5b by setting
includeCallTraces: IncludeTraces.All in the provider's observability config.

Plugin Test Suite Results

hardhat-gas-reporter v2.3.0

Category Pass Fail Notes
Unit tests 16 5 Failures: missing API keys (CMC/Etherscan)
Integration tests 30 7 Failures: missing API keys + Alchemy token

Trace-format failures: 0. All gas measurement and reporting functionality works.

solidity-coverage v0.8.17

Category Pass Fail Notes
Unit tests (before fix) 60 75 All failures: 0% coverage
Unit tests (after fix) 135 0 All recovered
Integration tests (before fix) 19 23 All failures: 0% coverage
Integration tests (after fix) 41 1 1 pre-existing (block gas limit)

Trace-format failures after fix: 0.

hardhat-tracer v3.4.0

Category Pass Fail Notes
Tests 9 4 Failures: 2 missing Alchemy key, 2 stdin bug

Trace-format failures: 0. All trace-exercising tests pass (CALLs, STATICCALLs,
DELEGATECALLs, opcodes, debug_traceTransaction).

Third-Party Project Results (with linked plugins)

Seaport (best result — full validation)

Mode Pass Fail Extra
Tests 402 0 --
Gas Reporter 402 0 Real gas data (e.g., ConduitController.createConduit avg 223,418)
Coverage 402 0 13.79% stmts overall; 95% for zones

OpenZeppelin Contracts v5.5.0

Mode Pass Fail Extra
Tests (ERC20) 126 0 --
Gas Reporter 126 0 Real gas data (approve avg 44,656; transfer avg 40,133)
Coverage -- -- Compilation fails (instrumentation bloat, not trace-related)

NexusMutual

Mode Pass Fail Extra
Tests 0 2 Gas cap issue (30M > 16.7M cap)
Gas Reporter 0 2 Same gas cap issue
Coverage 2 0 Non-zero data (Pool.sol: 29.77% stmts)

Additional projects tested (stock plugins, local Hardhat+EDR)

Project Tests Gas Reporter Coverage Trace Errors
Rocket Pool Subset pass Works Works None
Synthetix 99/99 pass Works N/A None
Safe Contracts 0/58 (gas cap) N/A N/A None

Non-Trace Issues Found

  1. EIP-7825 gas cap (Safe, NexusMutual): Transactions exceeding 16,777,216 gas
    fail. This is a hardfork behavioral change, not a trace issue.
  2. Synthetix function overloading: Pre-existing EDR limitation. Unrelated to traces.
  3. OpenZeppelin coverage compilation: solidity-coverage instrumentation bloat
    causes compilation failure on the full OZ codebase. Not trace-related.

Risk Assessment

Risk of trace format changes breaking existing Hardhat plugins: LOW

With the includeCallTraces: IncludeTraces.All fix in place, the backward-compatibility
layer in EDR (Response::traces()) combined with the Hardhat adapter successfully
preserves the expected trace format for all tested plugins:

  • hardhat-gas-reporter: Fully functional
  • solidity-coverage: Fully functional (after fix)
  • hardhat-tracer: Fully functional

@Wodann
Wodann requested a review from Copilot February 19, 2026 17:54
@Wodann Wodann self-assigned this Feb 19, 2026
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 17:54 — with GitHub Actions Error
@changeset-bot

changeset-bot Bot commented Feb 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a0d5e90

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@nomicfoundation/edr Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reintroduces backwards compatibility for the Response::traces getter to support Hardhat 2's EthereumJS VM interface. The implementation converts the new CallTraceArena format back to the flat trace format that Hardhat 2 expects.

Changes:

  • Adds Response::traces() method that converts CallTraceArena to flat trace format compatible with Hardhat 2
  • Removes edr_tracing dependency from multiple crates
  • Deletes nested_tracer.rs and related code that are no longer needed
  • Re-enables verbose tracing tests for Hardhat 2 compatibility

Reviewed changes

Copilot reviewed 22 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
crates/edr_napi/src/trace.rs Implements conversion from CallTraceArena to flat trace format with Hardhat 2 compatibility quirks
crates/edr_napi/src/provider/response.rs Adds traces() method to Response
crates/edr_provider/src/observability.rs Updates tracing configuration to capture full stack snapshots
patches/hardhat@2.28.4.patch Updates Hardhat 2 integration to use new trace format
crates/edr_napi/test/provider.ts Re-enables verbose tracing tests
Multiple Cargo.toml files Removes edr_tracing dependency
crates/edr_solidity/src/nested_tracer.rs Deleted - no longer needed
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/edr_napi/src/trace.rs Outdated
Comment on lines +149 to +154
let tracing_config = if config.verbose_raw_tracing {
TracingInspectorConfig::all()
} else {
TracingInspectorConfig::default_parity().set_steps(true)
TracingInspectorConfig::default_parity()
.set_steps(true)
.set_stack_snapshots(StackSnapshotType::Full)

Copilot AI Feb 19, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The StackSnapshotType::Full configuration is set regardless of verbose mode. This means that the full stack will always be captured in the tracing inspector, even when verbose tracing is disabled.

However, the actual filtering to return only the top of the stack happens in the conversion function raw_trace_from_call_trace_arena based on the verbose parameter. This creates an inconsistency where:

  1. The inspector always captures full stack (performance cost)
  2. The conversion then throws away most of the data when verbose=false

Consider setting StackSnapshotType based on verbose_raw_tracing to avoid unnecessary data collection and improve performance.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fixed in the stacked PRs rather than here: #1536 drops StackSnapshotType::Full from the non-verbose config and captures only the top of the stack per step.

@Wodann
Wodann force-pushed the feat/tracing-back-compat branch from 87edb00 to 041399f Compare February 19, 2026 18:01
@Wodann
Wodann temporarily deployed to github-action-benchmark February 19, 2026 18:01 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:08 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:08 — with GitHub Actions Error
@Wodann
Wodann temporarily deployed to github-action-benchmark February 19, 2026 18:08 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:16 — with GitHub Actions Failure
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:16 — with GitHub Actions Failure
@codecov

codecov Bot commented Feb 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.25123% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.10%. Comparing base (d35fa0e) to head (a0d5e90).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/edr_napi/src/trace.rs 86.23% 13 Missing and 2 partials ⚠️
crates/edr_napi/src/context.rs 63.88% 11 Missing and 2 partials ⚠️
crates/edr_napi/src/config.rs 80.00% 2 Missing ⚠️
crates/edr_provider/src/observability.rs 77.77% 2 Missing ⚠️
crates/edr_napi/src/async_deallocator.rs 87.50% 0 Missing and 1 partial ⚠️
crates/edr_provider/src/data.rs 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1301      +/-   ##
==========================================
+ Coverage   79.68%   80.10%   +0.42%     
==========================================
  Files         446      445       -1     
  Lines       76660    76479     -181     
  Branches    76660    76479     -181     
==========================================
+ Hits        61087    61267     +180     
+ Misses      13458    13088     -370     
- Partials     2115     2124       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Wodann
Wodann requested a review from a team February 20, 2026 19:49
@Wodann
Wodann marked this pull request as ready for review February 20, 2026 22:32

@popescuoctavian popescuoctavian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, but I believe there is an issue which Copilot pointed out and a small simplification.

Comment thread crates/edr_napi/src/trace.rs Outdated
Comment thread crates/edr_napi/src/trace.rs Outdated
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 14:59 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 15:07 — with GitHub Actions Inactive

@popescuoctavian popescuoctavian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is significant performance regression seen in the benchmarks -- up to 2.45x.

I believe this is caused by the inclusion of the full stack snapshot. However, I don't see an immediate solution here, with the existing revm-inspectors tracing code.

@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 19:23 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 19:40 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 19:40 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 20:53 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 21:29 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian force-pushed the feat/tracing-back-compat branch from d203b12 to 369dc66 Compare February 23, 2026 23:17
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 23:17 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 23:19 — with GitHub Actions Inactive
nebasuke added 2 commits July 6, 2026 10:34
The verbose-mode trace tests send raw eth_sendTransaction requests without
a gas field, so they inherited defaultTransactionGasLimit. Under the latest
L1 hardfork the EIP-7825 per-transaction gas cap rejects that, producing no
trace arena and an empty traces() result. Set an explicit sub-cap gas limit,
matching the existing isStaticCall test.
Regenerate the hardhat patch against 2.28.6: all code hunks apply
unchanged; hardhat#7918 only shifted the provider source maps, which are
spliced from the old patch and drift by a few mapping lines after the
changed region (cosmetic, dev-only).

Set observability.includeCallTraces: All at provider creation. Without
it Response.traces() returns no arenas and the re-enabled vm
trace-event bridge silently emits nothing (the solidity-coverage
0%-coverage failure mode).

Add a hardhat-tests spec asserting step/beforeMessage/afterMessage fire
on a transaction: the only executable coverage of the bridge.
@nebasuke
nebasuke force-pushed the feat/tracing-back-compat branch from 9acaaac to a8ef4d2 Compare July 6, 2026 11:12
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 6, 2026 11:12 — with GitHub Actions Inactive
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 6, 2026 12:14 — with GitHub Actions Failure
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 6, 2026 12:14 — with GitHub Actions Failure
@nebasuke
nebasuke had a problem deploying to github-action-benchmark July 6, 2026 20:56 — with GitHub Actions Failure
@Wodann

Wodann commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

I locally ran benchmarks comparing the state before the old TracingCollector was removed (#1278) with the performance of this PR—after Bas' fix (41931b5):

scenario baseline main@46b2d018a pre (old TraceCollector) @ f4bdc366b fixed @ 1379aaa1d pre/base fix/base fix/pre
neptune-mutual 23.72s 28.88s 25.26s 1.22 1.06 0.87
openzeppelin 12.10s 15.41s 15.15s 1.27 1.25 0.98
rocketpool 24.05s 23.94s 26.67s 1.00 1.11 1.11
safe-contracts 0.88s 0.94s 0.99s 1.07 1.13 1.06
seaport 5.85s 6.65s 7.50s 1.14 1.28 1.13
synthetix 310.01s 372.04s 346.17s 1.20 1.12 0.93
uniswap-v3-core 4.49s 5.49s 5.72s 1.22 1.28 1.04
TOTAL 381.10s 453.34s 427.46s 1.19 1.12 0.94

There is one more optimisation worth to avoid regressions in HH3; i.e. to never collect stack trace tops.

Wodann added 2 commits July 8, 2026 05:30
Also ignores `verbose` option when generating raw traces, since the
`ObservalityConfig` already has the `record_memory` and `record_stack`
options to control the level of detail in the traces.
@Wodann
Wodann temporarily deployed to github-action-benchmark July 8, 2026 05:41 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark July 8, 2026 05:44 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark July 8, 2026 05:44 — with GitHub Actions Error
@Wodann
Wodann requested a review from popescuoctavian July 8, 2026 05:44
@Wodann

Wodann commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

@popescuoctavian could you please review these changes that I made:

@Wodann
Wodann temporarily deployed to github-action-benchmark July 8, 2026 05:54 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark July 8, 2026 05:56 — with GitHub Actions Failure
@Wodann
Wodann temporarily deployed to github-action-benchmark July 8, 2026 05:56 — with GitHub Actions Inactive

@popescuoctavian popescuoctavian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me, thanks @Wodann!

I've noticed some benchmark regressions: https://github.com/NomicFoundation/edr/actions/runs/28920891714/job/85797979674?pr=1301. Are these expected?

@Wodann
Wodann temporarily deployed to github-action-benchmark July 8, 2026 14:35 — with GitHub Actions Inactive
@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​hardhat@​2.28.692100929580

View full report

@Wodann
Wodann had a problem deploying to github-action-benchmark July 8, 2026 14:37 — with GitHub Actions Failure
@Wodann
Wodann temporarily deployed to github-action-benchmark July 8, 2026 14:37 — with GitHub Actions Inactive
@Wodann

Wodann commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

I've noticed some benchmark regressions: https://github.com/NomicFoundation/edr/actions/runs/28920891714/job/85797979674?pr=1301. Are these expected?

@popescuoctavian I performed profiling to analyse the HH2 performance regression. Here's the outcome:

scenario pre-switch (old TraceCollector) f4bdc366b (CI, 2026-02-11) main latest b68e9664a / edr-0.14.1 (CI, 2026-07-08) PR feat/tracing-back-compat @ cb1a6c586 (CI run 28920891714) PR/pre PR/main
neptune-mutual 19.23s 22.41s 25.63s 1.33 1.14
openzeppelin 9.50s 9.48s 11.47s 1.21 1.21
rocketpool 11.30s 12.41s 14.80s 1.31 1.19
safe-contracts 0.65s 0.66s 0.72s 1.11 1.09
seaport 5.14s 4.90s 6.00s 1.17 1.22
synthetix 151.46s 151.07s 173.54s 1.15 1.15
uniswap-v3-core 3.93s 4.12s 4.93s 1.25 1.20
All Scenarios 201.21s 205.05s 237.08s 1.18 1.16

CI (run 28920891714) shows this branch at cb1a6c586 costs 1.13–1.16× vs main on the provider benchmarks. At this commit the HH2 patch sets recordStack: StackSnapshotType.Top + includeCallTraces: All, which is the same configuration profiled locally (perf, frame-pointer builds, seaport scenario), where the regression reproduces at 1.12× overall vs main@46b2d018a — same regression, same cause.

Where the time goes (seaport: 5.85s → 7.50s, +1.65s, +28%)

EVM/provider execution itself is unchanged — the entire delta is in producing, delivering, holding, and destroying per-response trace arenas:

mechanism baseline main@46b2d018a with traces (Top, includeCallTraces: All) Δ
V8/node native (GC + napi object traffic) 2.3s 3.3s +1.0s
arena teardown at napi finalize (drop_in_place<Response>Vec<CallTraceNode>mi_free) ~0 0.5s +0.5s
JIT'd JS (vm-trace bridge consuming steps) 1.7s 1.9s +0.2s
per-step recording (start_step: Top Box<[U256]> alloc + ~200-byte CallTraceStep + ordering push + step_end fill) 0.6s 0.7s +0.1s

The unifying driver is the first two rows, and the memory data explains them: peak RSS goes 445 MB → 1.5 GB (3.3×). Every Response carries its CallTraceArena and keeps it alive until the JS garbage collector finalizes the Response object. That costs twice: the V8 heap works against a much larger resident set (more GC time, worse locality — the +1.0s), and when finalization eventually runs, the arena teardown executes on the node main thread via napi raw_finalize (the +0.5s), serialized with request dispatch.

For scale: the pre-fix build (StackSnapshotType::Full, cloning the whole EVM stack per step) had the same lifetime pattern with bigger payloads — teardown 1.2s, RSS 5.6 GB, and OOM-killed on the openzeppelin scenario on a 15 GB machine.

Methodology
  • Machine: 24-core WSL2 devcontainer, 15 GB RAM. Builds: napi-publish profile with -Cforce-frame-pointers=yes (pnpm build:perf-js).
  • Timings: full pnpm providerBenchmark suite per build (harness medians of 3–15 fresh-subprocess runs per scenario), run sequentially on an idle machine.
  • Profiles: perf record -F 997 -g on single-scenario replays; flamegraphs and folded-stack aggregation via inferno; peak RSS via /usr/bin/time -v; mechanism attribution = share of samples whose stack matches the mechanism × median wall time.

I attempted a fix for the GC & teardown time of the Response object, by off-loading it to a background thread. This is similar to what we did for Provider. I'm awaiting benchmark results.

@Wodann

Wodann commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

Off-loading deallocation to a background thread resulted in better performance, but not on par with main yet:

scenario pre-switch (old TraceCollector) f4bdc366b (CI, 2026-02-11) main latest b68e9664a / edr-0.14.1 (CI, 2026-07-08) PR @ a0d5e90d7 (background-thread dealloc) PR/pre PR/main
neptune-mutual 19.23s 22.41s 23.02s 1.20 1.03
openzeppelin 9.50s 9.48s 10.84s 1.14 1.14
rocketpool 11.30s 12.41s 13.60s 1.20 1.10
safe-contracts 0.65s 0.66s 0.73s 1.12 1.10
seaport 5.14s 4.90s 5.33s 1.04 1.09
synthetix 151.46s 151.07s 166.18s 1.10 1.10
uniswap-v3-core 3.93s 4.12s 4.46s 1.13 1.08
All Scenarios 201.21s 205.05s 224.15s 1.11 1.09

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-enable traces for Hardhat 2

4 participants