Skip to content

feat(simq): add search statistics#2418

Merged
LHT129 merged 1 commit into
antgroup:mainfrom
LHT129:opencode/simq-statistics
Jul 9, 2026
Merged

feat(simq): add search statistics#2418
LHT129 merged 1 commit into
antgroup:mainfrom
LHT129:opencode/simq-statistics

Conversation

@LHT129

@LHT129 LHT129 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Populate search statistics for SIMQ KNN and range search result datasets.
  • Add SIMQ-specific counters for coarse search, rerank candidates, filtered candidates, result count, and range limited-size truncation.
  • Add regression coverage for SIMQ statistics output.

Fixes #2417

Test Plan

  • make fmt
  • cmake --build ./build/ --target functests --parallel 6
  • ./build/tests/functests -d yes [simq] --allow-running-no-tests
  • git diff --check

Copilot AI review requested due to automatic review settings July 6, 2026 07:18
@LHT129 LHT129 requested a review from wxyucs as a code owner July 6, 2026 07:18
@LHT129 LHT129 added the kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 label Jul 6, 2026
@LHT129 LHT129 requested a review from inabao as a code owner July 6, 2026 07:18
@LHT129 LHT129 requested a review from jiaweizone as a code owner July 6, 2026 07:18
@vsag-bot

vsag-bot commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

/label S-ready-to-merge
/waiting-on maintainer
/request-review @jiaweizone
/request-review @inabao

@mergify

mergify Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Require kind label

  • label~=^kind/

🟢 Require version label

  • label~=^version/

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive search statistics tracking for the SIMQ algorithm, including coarse distance comparisons, probe counts, candidate counts, and filtered candidates, which are serialized to JSON and attached to the result datasets. Corresponding unit tests have been added to validate these statistics. Feedback is provided regarding a potential null pointer dereference in the newly added read_dist_cmp helper function if the input dataset is null.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/algorithm/simq/simq.cpp

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds SIMQ-specific search statistics to result datasets (KNN + range) so SIMQ behavior can be observed and compared consistently with other indexes that already populate Dataset::Statistics.

Changes:

  • Extend SIMQ coarse search to accumulate coarse-stage probe and distance-comparison counters.
  • Populate SIMQ-specific statistic fields (e.g., coarse/rerank/filtered/result counts and limited-size truncation flag) into the returned dataset statistics JSON.
  • Add regression assertions in SIMQ tests to require statistics presence and basic invariants.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
tests/test_simq.cpp Adds helpers to parse/require SIMQ statistics and asserts stats are present in KNN/range tests.
src/algorithm/simq/simq.h Extends coarse_search signature to optionally return coarse-stage counters.
src/algorithm/simq/simq.cpp Implements SIMQ statistics dumping and wires coarse/rerank/filter/result counters into KNN + range search outputs (including empty-index results).

Comment thread tests/test_simq.cpp Outdated
Comment thread tests/test_simq.cpp Outdated
Comment thread tests/test_simq.cpp
@LHT129 LHT129 self-assigned this Jul 6, 2026
@LHT129 LHT129 force-pushed the opencode/simq-statistics branch from d5cb70a to ac37cfe Compare July 7, 2026 06:24
Populate Dataset statistics for SIMQ KNN and range search results, including coarse-search and rerank counters for observability.

Fixes antgroup#2417

Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Assisted-by: OpenCode:codex-1/gpt-5.5
@LHT129 LHT129 force-pushed the opencode/simq-statistics branch from ac37cfe to 1f8f826 Compare July 8, 2026 02:45
Copilot AI review requested due to automatic review settings July 8, 2026 02:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

tests/test_simq.cpp:316

  • The new statistics assertions don’t cover the empty-index path. Since this PR explicitly changes SIMQ to return non-empty statistics even when total_count_ == 0 / rep_hgraph_ == nullptr, a regression test should verify that KnnSearch on an unbuilt/empty SIMQ index still returns a dataset whose statistics JSON contains the expected SIMQ keys with zero counts (instead of "{}").
// ─────────────────────────────────────────────────────────────────────────────
// Test cases
// ─────────────────────────────────────────────────────────────────────────────

Comment thread src/algorithm/simq/simq.cpp

@wxyucs wxyucs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@LHT129 LHT129 merged commit 275a7e4 into antgroup:main Jul 9, 2026
20 checks passed
@LHT129 LHT129 deleted the opencode/simq-statistics branch July 9, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 module/testing size/L version/1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIMQ search should expose statistics

4 participants