Skip to content

feat(bench): add CodSpeed performance benchmarks - #117

Merged
samzong merged 1 commit into
mainfrom
codspeed-wizard-1785688654668
Aug 3, 2026
Merged

feat(bench): add CodSpeed performance benchmarks#117
samzong merged 1 commit into
mainfrom
codspeed-wizard-1785688654668

Conversation

@codspeed-hq

@codspeed-hq codspeed-hq Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

What

Adds continuous performance measurement with CodSpeed: a divan benchmark target covering Recall's session pipeline, and a GitHub Actions workflow that runs it in CPU simulation mode on every push and pull request.

Benchmarks

benches/recall.rs is a single divan target, grouped by pipeline stage (22 benchmarks total):

Group Covers
parsing Claude Code and Codex JSONL transcript parsing (8/64/256 turns, with and without the tool-call event stream)
indexing session and message writes into SQLite, including FTS5 index maintenance
search FTS5 keyword search, hybrid FTS + sqlite-vec search, JSONL export
analytics usage dashboard aggregation (1k/10k events), embedding text build, git remote normalization
rendering plain-text transcript and shareable HTML rendering (32/256 messages)

Fixtures are generated by a small deterministic xorshift generator, so the same synthetic transcripts, sessions and usage events are produced on every run — a change in measurement means a change in code, not in data. Assistant turns include prose, lists and fenced code blocks so the markdown renderer sees realistic input.

How internals are reached

Every module in this crate is pub(crate), and src/lib.rs intentionally exposes only init() and run(). Rather than widening visibility across the codebase, benchmarks sit behind a new optional bench feature that compiles src/bench_api.rs as pub mod bench_api. That module builds the fixtures and calls the real production code.

  • Default builds, make check and the shipped binary are unaffected: the bench target declares required-features = ["bench"], so cargo clippy --all-targets and cargo test --workspace behave exactly as before.
  • Four #[cfg(test)] helpers (Store::open_in_memory, SessionTopologyWrite::none) are now #[cfg(any(test, feature = "bench"))], and two adapter parse functions plus src/share/{meta,render} became pub(crate). No item is public without the feature.
  • Fixtures are side-effect free: transcripts are written to a TempDir and the index is an in-memory SQLite database, so running benchmarks never touches a real recall.db. No network access and no model download — the embedding provider is never constructed.

The exception to the visibility rule is documented in AGENTS.md, and the benchmark layout in DEVELOPMENT.md.

CI

.github/workflows/codspeed.yml follows the existing CI conventions (major-version tags, ubuntu-latest, push to main + pull_request):

  • simulation mode, the recommended instrument for Rust: deterministic CPU measurement, so results do not depend on runner noise.
  • cargo codspeed build --features bench is used instead of cargo build, since it adds the instrumentation CodSpeed needs.
  • workflow_dispatch is enabled so CodSpeed can trigger backtest runs and build the initial baseline.
  • [profile.bench] lto = "thin" keeps benchmark builds reasonable in CI; the release profile keeps full LTO.

Local verification

Everything below was run before opening this PR:

  • cargo codspeed build --features bench -m simulation — builds in ~2.5 min
  • codspeed run --mode simulation -- cargo codspeed run — all 22 benchmarks measured
  • make check — fmt, clippy -D warnings (--all-targets, and again with --features bench), 382 tests all pass

A make bench target was added so the same flow is one command locally.

Next steps

  1. Merge this PR. The first run on main becomes the performance baseline; later pull requests get a comparison report as a PR comment.
  2. Optionally trigger the workflow manually (workflow_dispatch) on a few past commits to backfill history.
  3. Extend coverage by adding a fixture to src/bench_api.rs and wiring it into the matching group in benches/recall.rs — for example the remaining adapters (Cursor, OpenCode, Gemini) or the semantic indexing path.

Add a divan benchmark target covering the session pipeline (transcript
parsing, SQLite indexing, keyword/hybrid search, export, usage
aggregation, transcript and share rendering) plus a CodSpeed workflow
that runs it in CPU simulation mode on every push and pull request.

Benchmarks live behind the optional `bench` feature, which exposes
`recall::bench_api` so the bench binary can reach crate-private hot paths
and build deterministic fixtures. Default builds and the shipped binary
are unchanged; fixtures use temp dirs and in-memory SQLite only.

Signed-off-by: CodSpeed Bot <no-reply@codspeed.io>
@codspeed-hq

codspeed-hq Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Congrats! CodSpeed is installed 🎉

🆕 22 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Open in CodSpeed

@codspeed-hq
codspeed-hq Bot marked this pull request as ready for review August 2, 2026 17:03
@codspeed-hq
codspeed-hq Bot requested a review from samzong as a code owner August 2, 2026 17:03
@samzong
samzong merged commit 29bbdde into main Aug 3, 2026
3 checks passed
@samzong
samzong deleted the codspeed-wizard-1785688654668 branch August 3, 2026 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants