feat(bench): add CodSpeed performance benchmarks - #117
Merged
Conversation
Add a divan benchmark target covering the session pipeline (transcript parsing, SQLite indexing, keyword/hybrid search, export, usage aggregation, transcript and share rendering) plus a CodSpeed workflow that runs it in CPU simulation mode on every push and pull request. Benchmarks live behind the optional `bench` feature, which exposes `recall::bench_api` so the bench binary can reach crate-private hot paths and build deterministic fixtures. Default builds and the shipped binary are unchanged; fixtures use temp dirs and in-memory SQLite only. Signed-off-by: CodSpeed Bot <no-reply@codspeed.io>
Contributor
Author
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds continuous performance measurement with CodSpeed: a divan benchmark target covering Recall's session pipeline, and a GitHub Actions workflow that runs it in CPU simulation mode on every push and pull request.
Benchmarks
benches/recall.rsis a single divan target, grouped by pipeline stage (22 benchmarks total):parsingindexingsearchanalyticsrenderingFixtures are generated by a small deterministic xorshift generator, so the same synthetic transcripts, sessions and usage events are produced on every run — a change in measurement means a change in code, not in data. Assistant turns include prose, lists and fenced code blocks so the markdown renderer sees realistic input.
How internals are reached
Every module in this crate is
pub(crate), andsrc/lib.rsintentionally exposes onlyinit()andrun(). Rather than widening visibility across the codebase, benchmarks sit behind a new optionalbenchfeature that compilessrc/bench_api.rsaspub mod bench_api. That module builds the fixtures and calls the real production code.make checkand the shipped binary are unaffected: the bench target declaresrequired-features = ["bench"], socargo clippy --all-targetsandcargo test --workspacebehave exactly as before.#[cfg(test)]helpers (Store::open_in_memory,SessionTopologyWrite::none) are now#[cfg(any(test, feature = "bench"))], and two adapter parse functions plussrc/share/{meta,render}becamepub(crate). No item is public without the feature.TempDirand the index is an in-memory SQLite database, so running benchmarks never touches a realrecall.db. No network access and no model download — the embedding provider is never constructed.The exception to the visibility rule is documented in
AGENTS.md, and the benchmark layout inDEVELOPMENT.md.CI
.github/workflows/codspeed.ymlfollows the existing CI conventions (major-version tags,ubuntu-latest,pushtomain+pull_request):simulationmode, the recommended instrument for Rust: deterministic CPU measurement, so results do not depend on runner noise.cargo codspeed build --features benchis used instead ofcargo build, since it adds the instrumentation CodSpeed needs.workflow_dispatchis enabled so CodSpeed can trigger backtest runs and build the initial baseline.[profile.bench] lto = "thin"keeps benchmark builds reasonable in CI; the release profile keeps full LTO.Local verification
Everything below was run before opening this PR:
cargo codspeed build --features bench -m simulation— builds in ~2.5 mincodspeed run --mode simulation -- cargo codspeed run— all 22 benchmarks measuredmake check— fmt, clippy-D warnings(--all-targets, and again with--features bench), 382 tests all passA
make benchtarget was added so the same flow is one command locally.Next steps
mainbecomes the performance baseline; later pull requests get a comparison report as a PR comment.workflow_dispatch) on a few past commits to backfill history.src/bench_api.rsand wiring it into the matching group inbenches/recall.rs— for example the remaining adapters (Cursor, OpenCode, Gemini) or the semantic indexing path.