Skip to content

fix(kpi): bind strict provenance to exact log bytes - #105

Closed
seonghobae wants to merge 27 commits into
mainfrom
fix/kpi-provenance-integrity-on-main
Closed

fix(kpi): bind strict provenance to exact log bytes#105
seonghobae wants to merge 27 commits into
mainfrom
fix/kpi-provenance-integrity-on-main

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Purpose

Test-first repair for #104: strict production KPI evidence must be bound to the exact retained exchange-30d.ndjson bytes that are scored. The gate authenticates exact bytes, scores a verified private snapshot rather than a mutable original path, derives provenance record count/digest/byte size from one retained-byte stream, and fails closed when an explicitly configured strict evidence sink cannot retain the result.

Exact source identity

  • protected base: main at c85d710804139c0697d7ef8fa47d02b1389e6d84;
  • head branch: fix/kpi-provenance-integrity-on-main;
  • original provenance-integrity RED: a8fc068f0b3a2bdad3a72db4d3f6018dfaf18f20;
  • first digest/byte-binding GREEN: e12e41d33b1b0d2cea9dfbb42550e94f850cac7b;
  • replace-score-restore TOCTOU RED: b9c51582ab701f6851ee65e0f9283e90f5c6e53f;
  • verified-snapshot GREEN: e80c6a8541b480cd7c381c7042ea9e233863dd20;
  • unterminated-NDJSON provenance-count RED: 676f31190226f1e9ebf566fc0914eb2666efd257;
  • same-stream record/digest/byte GREEN: 61b8c152698d2833fda6dfbb4a1b0dc951f468cd;
  • strict evidence-persistence RED: 807a867600249a1101e52965de47ed81f77e48e1;
  • strict evidence-persistence GREEN: b4055f30bbe1d93891cba04d909b183e6503e505;
  • current exact head after positive-safe-integer logBytes mismatch regression coverage: 825dd9b8c3f19270602735a0c78a417fe7213345.

RCA -> RED -> GREEN

1. Provenance-to-evidence identity

Collection originally retained source metadata but did not authenticate the exact scored bytes. The repair records exact SHA-256 plus byte size and validates both before scoring.

2. Replace-score-restore TOCTOU

A start/end identity comparison was insufficient: a process could replace the original log after initial verification, allow KPI child checks to score different bytes, and restore the original before the final identity check.

The deterministic RED at b9c51582ab701f6851ee65e0f9283e90f5c6e53f uses a Node preload hook only in KPI child processes to replace the original log with a failing log and restore it on child exit. Application CI run 31414867826, job 93541377446, checked out that exact head and failed exactly the new snapshot-race regression; the other 652 tests passed.

The narrow GREEN validates original SHA-256 and positive safe-integer byte size, copies verified source bytes into a process-private temporary snapshot, sets mode 0400, recomputes snapshot identity, passes only the verified snapshot to KPI child checks, retains a final original-path identity check, and removes temporary bytes on normal completion/process exit. This stabilizes scored bytes against original-path replacement/restoration without claiming an OS privilege boundary against same-account/higher-privilege tampering.

3. Record-count/data-integrity boundary

Fresh collector inspection found shell wc -l computed records before a separate Node pass computed logSha256/logBytes. An unterminated final NDJSON record therefore produced records: 0 even though one valid record existed, and count versus digest/byte metadata came from separate reads.

Exact RED 676f31190226f1e9ebf566fc0914eb2666efd257 / application CI 31415905532, job 93544736430, failed exactly the count regression while 653 tests passed. GREEN 61b8c152698d2833fda6dfbb4a1b0dc951f468cd removes wc -l authority and derives records, logSha256, and logBytes from one retained-byte stream, including a final non-empty unterminated record.

4. Strict evidence-retention boundary

Fresh exact-head inspection found persistEvidence() logged an NOEMA_KPI_EVIDENCE_PATH write error but swallowed it, so strict mode could still exit 0 and emit PASS while explicitly configured retained evidence did not exist.

RED 807a867600249a1101e52965de47ed81f77e48e1 configures the evidence destination as a directory so persistence deterministically fails. Application CI run 31417456928, job 93549720229, checked out that exact head and failed only the new regression (status was 0, expected 1); 654 other tests passed.

GREEN b4055f30bbe1d93891cba04d909b183e6503e505 makes evidence persistence return success/failure and requires successful retention before strict mode may emit final PASS. Non-strict optional-evidence behavior is unchanged; already-failing strict paths remain failing even if best-effort diagnostic evidence cannot be written.

5. Positive logBytes identity mismatch coverage

A diagnostic review of exact head b4055f30bbe1d93891cba04d909b183e6503e505 found a current regression-coverage gap: missing/zero/unsafe logBytes cases proved field validation, but no test kept a valid digest while supplying a different positive safe-integer byte size to prove the actual identity-mismatch path.

Current head 825dd9b8c3f19270602735a0c78a417fe7213345 adds that focused case. The provenance keeps the correct SHA-256, increments logBytes by one, and requires strict failure with KPI log identity does not match production provenance. No production implementation change was needed because the existing exact identity check already fails closed.

Current exact-head technical evidence

For unchanged exact head 825dd9b8c3f19270602735a0c78a417fe7213345:

  • application ci run 31418085423: terminal success;
  • reviewer-ci run 31418085451: terminal success;
  • central Security Scan run 31418085400: terminal success under current protected-base scanner semantics;
  • the two original CodeRabbit inline findings remain addressed/resolved; current unresolved inline threads: 0;
  • the diagnostic P2 regression-coverage finding on predecessor b4055f30bbe1d93891cba04d909b183e6503e505 is addressed by current head 825dd9b8c3f19270602735a0c78a417fe7213345;
  • the visible formal CodeRabbit submission remains COMMENTED on much earlier predecessor 43054db5bc8486d56db77bd17f2371d633e678a5, not qualifying independent approval;
  • automatic model review of the current head is rate-limited/pending retry and therefore non-passing, non-approval evidence;
  • GitHub reports the content graph mergeable; branch is 16 commits ahead / 0 behind current protected main.

The ordinary release verification still treats absent real production KPI input as non-operational evidence. Acquisition evidence remains fail closed where real production, release/deployment, customer/revenue, owner/legal and transfer evidence is absent.

Evidence boundary

This change authenticates and stably scores the exact retained bytes supplied to the KPI gate, keeps provenance record/digest/byte metadata coherent for one retained-byte stream, prevents configured strict evidence-retention failure from producing a successful process result, and explicitly tests a valid-but-wrong byte count. It does not authenticate the upstream Logpush/archive provider, prove 30 days of real production operation, establish deployment/release state, provide customer/revenue evidence, choose an outbound license, establish owner/legal/IP-transfer authority, or turn the temporary snapshot into an OS privilege boundary.

Merge authority

Technical exact-head success is not merge authority. Live enforceable governance under #27 and the qualifying independent non-author formal-review path under #29 remain fail-closed where required. No self-approval, protection weakening, synthetic status, release, deployment or acquisition-readiness claim is made.

Related: #3, #5, #27, #29, #104

Summary by CodeRabbit

  • 개선 사항

    • KPI 로그 수집 시 레코드 수, 파일 크기, SHA-256 무결성 정보가 정확히 기록됩니다.
    • 마지막 개행이 없는 NDJSON 로그도 올바르게 처리됩니다.
    • 엄격한 KPI 검증에서 로그 변경, 잘못된 provenance, 안전하지 않은 파일 경로를 감지합니다.
    • 검증 중 원본 로그가 교체되어도 검증된 스냅샷을 사용합니다.
  • 문서

    • KPI provenance 검증 및 외부 보관 절차가 최신 동작에 맞게 업데이트되었습니다.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 824fe11c-3eca-451d-9233-df0d2f9979d1

📥 Commits

Reviewing files that changed from the base of the PR and between 3d8a7da and b834107.

📒 Files selected for processing (4)
  • scripts/collect-kpi-logs.sh
  • scripts/kpi-gate.mjs
  • test/kpi-collect.test.ts
  • test/kpi-gate.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • test/kpi-collect.test.ts
  • scripts/kpi-gate.mjs

📝 Walkthrough

Walkthrough

KPI 로그 수집기가 NDJSON 레코드 수, logSha256, logBytes를 provenance에 기록합니다. strict KPI 게이트는 검증된 읽기 전용 스냅샷으로 검사하고 원본 로그 identity를 재검증합니다. 테스트와 운영 문서가 새 무결성 조건을 반영합니다.

Changes

KPI 로그 provenance 강화

Layer / File(s) Summary
로그 identity 생성
scripts/collect-kpi-logs.sh, test/kpi-collect.test.ts
수집기가 로그를 스트리밍하여 레코드 수, SHA-256 해시, 바이트 수를 계산합니다. 마지막 개행이 없는 NDJSON도 처리합니다. HTTPS 정책과 HTTP 실패 처리도 검증합니다.
strict KPI 게이트 검증
scripts/kpi-gate.mjs
strict 게이트가 provenance와 원본 로그 identity를 검증합니다. 검증된 읽기 전용 스냅샷으로 KPI 검사를 실행하고 검사 후 원본 변경을 확인합니다. evidence 저장 실패도 strict 실패로 처리합니다.
회귀 테스트와 운영 계약
test/kpi-gate.test.ts, test/kpi-provenance-snapshot.test.ts, docs/observability-kpi.md
로그 교체, identity 누락·오류, 안전하지 않은 logBytes, 심볼릭 링크 경로와 evidence 저장 실패를 검증합니다. 체크리스트와 외부 아카이브 절차에 실제 로그 identity 확인을 추가합니다.

Estimated code review effort: 4 (Complex) | ~45 minutes

Mergeability Score: ⚪ Minimal · up to b8341

The PR binds KPI results to exact retained log bytes and fails closed when strict evidence cannot be retained; no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant Collector as collect-kpi-logs.sh
  participant LogFile as KPI log file
  participant Provenance as provenance JSON
  participant Gate as kpi-gate.mjs
  participant Snapshot as read-only snapshot
  participant KPI as KPI checks
  Collector->>LogFile: 로그 스트리밍
  LogFile-->>Collector: 레코드와 바이트
  Collector->>Provenance: records, logSha256, logBytes 기록
  Gate->>Provenance: provenance 로드 및 형식 검증
  Gate->>LogFile: 로그 identity 계산
  LogFile-->>Gate: SHA-256 해시 및 바이트 수
  Gate->>Snapshot: 검증된 로그 복사
  Gate->>KPI: 스냅샷으로 KPI 검사 실행
  Gate->>LogFile: 최종 로그 identity 재계산
  Gate->>Provenance: identity 일치 여부 검증
Loading

Possibly related issues

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 strict KPI provenance를 정확한 로그 바이트에 연결하는 주요 변경 사항을 명확하고 간결하게 설명합니다.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/kpi-provenance-integrity-on-main

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae marked this pull request as ready for review August 10, 2026 17:22

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/kpi-gate.mjs`:
- Around line 116-143: Update the KPI execution flow around the initial
provenance identity check to create a permission-restricted temporary snapshot
after validating the original log identity, then compare the snapshot identity
with provenance and pass the snapshot path to every KPI subcheck instead of
logPath. Ensure cleanup handles the snapshot on success and failure, add a
regression test covering replacement and restoration of the original file during
checks, and revise the observability KPI documentation to describe the
snapshot-based file-change guarantee.

In `@test/kpi-gate.test.ts`:
- Around line 169-189: Extend the regression coverage around the “rejects
malformed strict provenance log identity” test using a valid lowercase
64-character logSha256, and add separate cases for missing logBytes, logBytes
set to 0, and a non-safe-integer value. Each case should invoke runKpiGate and
assert status 1 plus the exact “logBytes must be a positive safe integer”
message, while preserving the existing cleanup and protocol assertions.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4d97206d-478d-4cf9-b33d-54108a8463b4

📥 Commits

Reviewing files that changed from the base of the PR and between c85d710 and 43054db.

📒 Files selected for processing (5)
  • docs/observability-kpi.md
  • scripts/collect-kpi-logs.sh
  • scripts/kpi-gate.mjs
  • test/kpi-collect.test.ts
  • test/kpi-gate.test.ts

Comment thread scripts/kpi-gate.mjs
Comment thread test/kpi-gate.test.ts

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Please review the current exact head 41fa0f7b32e386203fefa19c823ca043c8495407, including the verified-snapshot TOCTOU repair and the independent logBytes regressions. Treat this as diagnostic review evidence only; it is not merge approval authority.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

expected='41fa0f7b32e386203fefa19c823ca043c8495407'
actual="$(git rev-parse HEAD)"
printf 'HEAD=%s\nEXPECTED=%s\n' "$actual" "$expected"
git status --short
git diff --check "${expected}^" "$expected"
git diff --find-renames --unified=80 "${expected}^" "$expected" -- \
  docs/observability-kpi.md \
  scripts/collect-kpi-logs.sh \
  scripts/kpi-gate.mjs \
  test/kpi-collect.test.ts \
  test/kpi-gate.test.ts

printf '\n== Source outlines ==\n'
ast-grep outline scripts/kpi-gate.mjs --items all
ast-grep outline test/kpi-gate.test.ts --items all
ast-grep outline test/kpi-collect.test.ts --items all

printf '\n== Identity-related references ==\n'
rg -n -C 5 -P '\b(logSha256|logBytes|loadProvenance|createHash|statSync|readFileSync|verified|snapshot)\b' \
  scripts/kpi-gate.mjs scripts/collect-kpi-logs.sh test/kpi-gate.test.ts test/kpi-collect.test.ts docs/observability-kpi.md

Length of output: 29804


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== kpi-gate strict-mode control flow =='
sed -n '1,230p' scripts/kpi-gate.mjs
sed -n '219,390p' scripts/kpi-gate.mjs

printf '%s\n' '== kpi-gate regression tests =='
sed -n '1,420p' test/kpi-gate.test.ts

printf '%s\n' '== collector and collector tests =='
sed -n '1,150p' scripts/collect-kpi-logs.sh
sed -n '1,180p' test/kpi-collect.test.ts

printf '%s\n' '== parent version for changed strict-gate behavior =='
git show '41fa0f7b32e386203fefa19c823ca043c8495407^:scripts/kpi-gate.mjs' | sed -n '1,430p'

printf '%s\n' '== references to strict gate invocation and provenance shape =='
rg -n -C 4 -P '\b(kpi-gate|kpi:verify:strict|NOEMA_KPI_STRICT|logSha256|logBytes)\b' \
  package.json scripts test .github 2>/dev/null || true

Length of output: 64781


🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='41fa0f7b32e386203fefa19c823ca043c8495407'

printf '%s\n' '== Pull request metadata =='
gh pr view 105 --repo ContextualWisdomLab/noema --json baseRefName,baseRefOid,headRefName,headRefOid,commits,files \
  --jq '{baseRefName,baseRefOid,headRefName,headRefOid, commits: [.commits[] | {oid,messageHeadline}], files: [.files[] | {path,additions,deletions}]}' \
  || true

base="$(git merge-base "$head" "origin/HEAD" 2>/dev/null || true)"
if [[ -z "$base" ]]; then
  base="$(git merge-base "$head" "origin/main" 2>/dev/null || true)"
fi
printf 'MERGE_BASE=%s\n' "$base"
if [[ -n "$base" ]]; then
  git diff --check "$base" "$head"
  git diff --name-status "$base" "$head"
  git diff --unified=100 "$base" "$head" -- test/kpi-provenance-snapshot.test.ts scripts/kpi-gate.mjs
fi

printf '%s\n' '== Dedicated snapshot and byte-size regressions =='
ast-grep outline test/kpi-provenance-snapshot.test.ts --items all
sed -n '1,360p' test/kpi-provenance-snapshot.test.ts

printf '%s\n' '== Test and coverage configuration =='
sed -n '1,220p' package.json
fd -HI 'vitest*' . --exclude node_modules --exclude .git -x sh -c 'echo "== $1 =="; sed -n "1,240p" "$1"' sh {}

Length of output: 30631


@seonghobae Diagnostic review of 41fa0f7b32e386203fefa19c823ca043c8495407 is complete. I found one regression-coverage gap.

P2 — Test a valid but mismatched logBytes value
test/kpi-provenance-snapshot.test.ts, Line 121 tests missing, zero, and unsafe logBytes values. These cases test field validation only. The changed-log test also changes logSha256, so it passes if the gate compares only the digest.

Add a case where provenance has the correct logSha256 but a different positive safe-integer logBytes. Expect strict mode to fail with KPI log identity does not match production provenance. This test proves that logBytes participates in the identity comparison.

The verified-snapshot flow is otherwise consistent with the documented boundary. Strict mode validates the original bytes, validates the private snapshot bytes, passes only the snapshot path to KPI child processes, and rechecks the original bytes after child execution. The snapshot-race regression correctly exercises replacement and restoration of the original log during child processing.

This is diagnostic review evidence only. It is not merge approval authority.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Please review the current exact head 61b8c152698d2833fda6dfbb4a1b0dc951f468cd. In addition to the verified-snapshot TOCTOU repair, review the collector change that derives records, logSha256, and logBytes from the same byte stream and correctly counts a final unterminated NDJSON record. Treat this as diagnostic model review evidence only; it is not merge approval authority.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='61b8c152698d2833fda6dfbb4a1b0dc951f468cd'

printf '%s\n' '== Exact revision and working tree =='
printf 'HEAD=%s\nREQUESTED=%s\n' "$(git rev-parse HEAD)" "$head"
git status --short
git cat-file -e "${head}^{commit}"
git diff --check "${head}^" "$head"

printf '%s\n' '== Changed files and patch =='
git diff --name-status "${head}^" "$head"
git diff --find-renames --unified=100 "${head}^" "$head" -- \
  scripts/collect-kpi-logs.sh \
  scripts/kpi-gate.mjs \
  test/kpi-collect.test.ts \
  test/kpi-gate.test.ts \
  test/kpi-provenance-snapshot.test.ts \
  docs/observability-kpi.md

printf '%s\n' '== Source maps =='
ast-grep outline scripts/kpi-gate.mjs --items all
ast-grep outline test/kpi-gate.test.ts --items all
ast-grep outline test/kpi-provenance-snapshot.test.ts --items all
ast-grep outline test/kpi-collect.test.ts --items all

printf '%s\n' '== Relevant definitions, invocation sites, and documentation =='
rg -n -C 5 -P '\b(logSha256|logBytes|records|loadProvenance|createHash|createReadStream|snapshot|verified|kpi-gate|collect-kpi-logs)\b' \
  scripts test package.json docs .github 2>/dev/null || true

Length of output: 170400


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== Strict gate execution and identity functions =='
sed -n '1,220p' scripts/kpi-gate.mjs
sed -n '219,385p' scripts/kpi-gate.mjs

printf '%s\n' '== Collector tests =='
sed -n '1,180p' test/kpi-collect.test.ts

printf '%s\n' '== Strict gate provenance tests =='
sed -n '1,360p' test/kpi-gate.test.ts

printf '%s\n' '== Snapshot integrity tests =='
sed -n '1,220p' test/kpi-provenance-snapshot.test.ts

printf '%s\n' '== Exact logBytes mismatch regression search =='
rg -n -C 6 -P '(logBytes\s*[:=].*(\+|\-)|logBytes.*identity does not match|identity does not match.*logBytes|mismatched.*logBytes|logBytes.*mismatch)' \
  test scripts docs 2>/dev/null || true

printf '%s\n' '== Commit ancestry and focused parent comparison =='
git show --no-patch --format='commit=%H%nauthor=%an%ndate=%cI%nparents=%P%nsubject=%s' HEAD
git diff --unified=80 '61b8c152698d2833fda6dfbb4a1b0dc951f468cd^' \
  '61b8c152698d2833fda6dfbb4a1b0dc951f468cd' -- test/kpi-collect.test.ts test/kpi-provenance-snapshot.test.ts scripts/kpi-gate.mjs

Length of output: 33917


@seonghobae Diagnostic review of 61b8c152698d2833fda6dfbb4a1b0dc951f468cd is complete.

I found one regression-coverage gap.

P2 — Add a valid mismatched logBytes regression
test/kpi-provenance-snapshot.test.ts, Line 117 validates only missing, zero, and unsafe logBytes values. These cases validate the field format. They do not prove that the strict identity comparison uses logBytes.

Add a case that retains the correct logSha256 but sets logBytes to a different positive safe integer. Expect strict mode to fail with:

KPI log identity does not match production provenance.

The changed-log test in test/kpi-gate.test.ts changes both the digest and byte size. A digest-only comparison would also fail that test.

The collector change is otherwise correct. scripts/collect-kpi-logs.sh derives records, logSha256, and logBytes during one createReadStream() traversal. It counts an unterminated final NDJSON record when the final byte is not LF. test/kpi-collect.test.ts verifies that result against the retained bytes.

The verified-snapshot flow is also consistent. Strict mode validates the original log identity, validates the copied read-only snapshot identity, passes only the snapshot path to KPI child processes, and rechecks the original identity after child execution.

This is diagnostic review evidence only. It is not merge approval authority.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Please review the unchanged current exact head 5b97a85004fe05e553150a63351d7b0a5a2cfc82. The source now uses a verified private snapshot for KPI child checks and derives records, logSha256, and logBytes from one retained-byte stream, including an unterminated final NDJSON record; operator documentation is aligned with that exact contract. Treat this as diagnostic model review evidence only, never as independent merge approval authority.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='5b97a85004fe05e553150a63351d7b0a5a2cfc82'

printf '%s\n' '== Exact revision =='
printf 'HEAD=%s\nREQUESTED=%s\n' "$(git rev-parse HEAD)" "$head"
git cat-file -e "${head}^{commit}"
git status --short
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' "$head"

printf '%s\n' '== PR metadata and merge-base diff =='
gh pr view 105 --repo ContextualWisdomLab/noema \
  --json baseRefOid,headRefOid,files \
  --jq '{baseRefOid,headRefOid,files:[.files[]|{path,additions,deletions}]}' || true
base="$(git merge-base "$head" origin/main 2>/dev/null || true)"
printf 'MERGE_BASE=%s\n' "$base"
if [[ -n "$base" ]]; then
  git diff --check "$base" "$head"
  git diff --name-status "$base" "$head"
  git diff --unified=100 "$base" "$head" -- \
    scripts/collect-kpi-logs.sh \
    scripts/kpi-gate.mjs \
    test/kpi-collect.test.ts \
    test/kpi-gate.test.ts \
    test/kpi-provenance-snapshot.test.ts \
    docs/observability-kpi.md
fi

printf '%s\n' '== Source outlines =='
ast-grep outline scripts/kpi-gate.mjs --items all
ast-grep outline test/kpi-collect.test.ts --items all
ast-grep outline test/kpi-gate.test.ts --items all
ast-grep outline test/kpi-provenance-snapshot.test.ts --items all

printf '%s\n' '== Focused source and tests =='
sed -n '1,240p' scripts/collect-kpi-logs.sh
sed -n '1,430p' scripts/kpi-gate.mjs
sed -n '1,260p' test/kpi-collect.test.ts
sed -n '1,420p' test/kpi-gate.test.ts
sed -n '1,300p' test/kpi-provenance-snapshot.test.ts
sed -n '1,240p' docs/observability-kpi.md

printf '%s\n' '== Targeted regression and invocation search =='
rg -n -C 5 -P '\b(logSha256|logBytes|records|snapshot|identity does not match|unterminated|createReadStream|createHash|kpi-gate)\b' \
  scripts test docs package.json .github 2>/dev/null || true

Length of output: 230239


@seonghobae Diagnostic review of exact head 5b97a85004fe05e553150a63351d7b0a5a2cfc82 is complete.

I found one regression-coverage gap.

P2 — Test a valid mismatched logBytes value
test/kpi-provenance-snapshot.test.ts, Line 117 tests missing, zero, and unsafe logBytes values. These cases test field validation only.

Add a case that preserves the correct logSha256 but sets logBytes to a different positive safe integer. Strict mode must fail with:

KPI log identity does not match production provenance.

This test proves that strict identity comparison uses logBytes. The existing changed-log test changes both fields. A digest-only comparison would also fail that test.

The implementation otherwise matches the stated contract:

  • The collector derives records, logSha256, and logBytes in one retained-byte stream traversal.
  • The collector counts a non-empty final NDJSON record without a trailing LF.
  • Strict mode verifies original-log identity, verifies the private snapshot identity, supplies only the snapshot path to KPI child checks, and rechecks original-log identity after those checks.
  • docs/observability-kpi.md describes these boundaries and limits.

Static inspection verified the requested exact head and passed git diff --check. I did not execute repository tests in this environment.

This is diagnostic review evidence only. It is not merge approval authority.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Please review the unchanged current exact head b4055f30bbe1d93891cba04d909b183e6503e505. In addition to the verified private snapshot and same-stream records/logSha256/logBytes provenance boundary, review the strict evidence-retention repair: an explicitly configured NOEMA_KPI_EVIDENCE_PATH write failure must prevent final strict PASS and exit nonzero. RED proof is exact head 807a867600249a1101e52965de47ed81f77e48e1 / application CI run 31417456928, and GREEN exact-head application CI 31417626486, reviewer-ci 31417626505, and central Security Scan 31417626428 are terminal success. Treat this as diagnostic model-review evidence only; it is not qualifying independent merge approval authority.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

head='b4055f30bbe1d93891cba04d909b183e6503e505'

printf '%s\n' '== Exact revision =='
printf 'HEAD=%s\nREQUESTED=%s\n' "$(git rev-parse HEAD)" "$head"
git cat-file -e "${head}^{commit}"
git status --short
git show --no-patch --format='commit=%H%nparents=%P%nsubject=%s' "$head"

printf '%s\n' '== PR metadata and merge-base change set =='
gh pr view 105 --repo ContextualWisdomLab/noema \
  --json baseRefOid,headRefOid,files \
  --jq '{baseRefOid,headRefOid,files:[.files[]|{path,additions,deletions}]}' || true
base="$(git merge-base "$head" origin/main 2>/dev/null || true)"
printf 'MERGE_BASE=%s\n' "$base"
if [[ -n "$base" ]]; then
  git diff --check "$base" "$head"
  git diff --name-status "$base" "$head"
  git diff --unified=100 "$base" "$head" -- \
    scripts/kpi-gate.mjs \
    scripts/collect-kpi-logs.sh \
    test/kpi-gate.test.ts \
    test/kpi-provenance-snapshot.test.ts \
    test/kpi-collect.test.ts \
    docs/observability-kpi.md
fi

printf '%s\n' '== Source maps =='
ast-grep outline scripts/kpi-gate.mjs --items all
ast-grep outline test/kpi-gate.test.ts --items all
ast-grep outline test/kpi-provenance-snapshot.test.ts --items all
ast-grep outline test/kpi-collect.test.ts --items all

printf '%s\n' '== Strict mode and evidence retention =='
rg -n -C 8 -P '\b(NOEMA_KPI_EVIDENCE_PATH|evidence|PASS|process\.exit|exitCode|writeFile|writeFileSync|loadProvenance|logSha256|logBytes|snapshot|records)\b' \
  scripts/kpi-gate.mjs scripts/collect-kpi-logs.sh test/kpi-gate.test.ts test/kpi-provenance-snapshot.test.ts test/kpi-collect.test.ts docs/observability-kpi.md
sed -n '1,460p' scripts/kpi-gate.mjs
sed -n '1,460p' test/kpi-gate.test.ts
sed -n '1,320p' test/kpi-provenance-snapshot.test.ts
sed -n '1,240p' test/kpi-collect.test.ts
sed -n '1,260p' docs/observability-kpi.md

printf '%s\n' '== Cited workflow runs =='
for run in 31417456928 31417626486 31417626505 31417626428; do
  gh run view "$run" --repo ContextualWisdomLab/noema \
    --json databaseId,headSha,workflowName,status,conclusion,event,url \
    --jq '{databaseId,headSha,workflowName,status,conclusion,event,url}' || true
done

Length of output: 156181


@seonghobae Diagnostic review of exact head b4055f30bbe1d93891cba04d909b183e6503e505 is complete.

I found one regression-coverage gap.

P2 — Test a valid mismatched logBytes value
test/kpi-provenance-snapshot.test.ts, Line 117 tests missing, zero, and unsafe logBytes values. These cases test field validation only.

Add a case that keeps the correct logSha256 but sets logBytes to a different positive safe integer. Strict mode must fail with:

KPI log identity does not match production provenance.

This test proves that strict identity comparison uses logBytes. The existing changed-log test changes both identity fields. A digest-only comparison would also fail that test.

The strict evidence-retention repair is correct by static inspection:

  • persistEvidence() returns false after a configured write failure.
  • In strict mode, scripts/kpi-gate.mjs exits with status 1 before it writes final PASS.
  • test/kpi-gate.test.ts uses a directory as NOEMA_KPI_EVIDENCE_PATH and verifies the nonzero exit and write-error output.

The existing provenance boundary remains consistent:

  • The collector derives records, logSha256, and logBytes from one retained-byte stream.
  • The collector counts an unterminated final NDJSON record.
  • Strict mode verifies the original log, verifies the private snapshot, gives child checks only the snapshot path, and then rechecks the original log.

Static inspection verified the requested head. git diff --check passed. I did not execute repository tests in this environment.

The cited application CI RED run 31417456928 failed on 807a867600249a1101e52965de47ed81f77e48e1. The application CI run 31417626486 and reviewer CI run 31417626505 succeeded on the requested head. I could not verify the cited central Security Scan because GitHub returned HTTP 404 for its workflow.

This is diagnostic review evidence only. It is not merge approval authority.

You are interacting with an AI system.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/kpi-gate.mjs`:
- Around line 376-383: Import constants from node:fs instead of node:crypto and
use fs.constants.O_NOFOLLOW in persistEvidence() when opening the evidence file.
Update the symlink test to preserve the target file and assert that the strict
execution exits with status 1.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 83d885f9-ae96-436c-ad66-2a715c46fb1d

📥 Commits

Reviewing files that changed from the base of the PR and between 43054db and 79846fa.

📒 Files selected for processing (6)
  • docs/observability-kpi.md
  • scripts/collect-kpi-logs.sh
  • scripts/kpi-gate.mjs
  • test/kpi-collect.test.ts
  • test/kpi-gate.test.ts
  • test/kpi-provenance-snapshot.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/collect-kpi-logs.sh

Comment thread scripts/kpi-gate.mjs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/kpi-collect.test.ts`:
- Line 136: Update both relevant tests in the provenance failure cases to also
assert that logPath does not exist, alongside the existing provenancePath
assertions. Preserve the current token non-disclosure and protocol-related
assertions while adding coverage for the absence of log files after URL
validation or HTTP failure.
- Line 135: Update the verify pipeline in scripts/collect-kpi-logs.sh so both
non-HTTPS URL and HTTP-error validation failures propagate exit status 1 to the
caller. Preserve the existing success status behavior and do not weaken the
assertions in test/kpi-collect.test.ts.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 146c25f5-9a88-4a7e-beb6-5f74dc50fdf0

📥 Commits

Reviewing files that changed from the base of the PR and between 79846fa and 3d8a7da.

📒 Files selected for processing (2)
  • scripts/kpi-gate.mjs
  • test/kpi-collect.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/kpi-gate.mjs

Comment thread test/kpi-collect.test.ts
Comment thread test/kpi-collect.test.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant