From 5f8273222c313553299d1787f8c0916cfc650944 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Fri, 14 Aug 2026 12:48:40 +0200 Subject: [PATCH] fix(ci): release notes report the shipped binary, not every candidate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Release notes now carry one row per product — the bytes actually published, with their VirusTotal verdict — instead of a column per candidate. Several candidates are scanned so the selector has an alternative when an opaque classifier flags one, but a reader installing cbm cares about what they receive, not about what we discarded. The rejected candidates' verdicts remain in the published evidence TSVs for anyone auditing the selection, and they stay useful to US as a development signal; in a changelog they are noise, and listing a "microsoft-ml" verdict next to a binary nobody can download invites exactly the wrong conclusion. Also removes a claim that had become FALSE: the section asserted that every scan had a minimum of 50 decisive engines. That floor was removed deliberately (it is VirusTotal fleet availability, not a property of our binary) and the last release observed 31. Published notes would have stated something untrue. The observed range is still reported, now over the shipped binaries. The disclosure of a tolerated Microsoft `!ml` is preserved and narrowed to the binaries that ship. The contract test now pins both directions: a flagged binary that SHIPS must be disclosed, and flagged candidates that were rejected must not appear at all. Its fixture gained a target that draws the tolerated verdict on all three candidates, so the disclosure branch is actually exercised — previously every flagged candidate was rejected in favour of a clean sibling, leaving that path untested. README and SECURITY.md updated: they promised notes "link every candidate result". All five release/VT contract tests pass. Signed-off-by: Martin Vogel --- README.md | 2 +- SECURITY.md | 2 +- scripts/ci/append-vt-notes.sh | 42 ++++++++++++++++--------- tests/test_vt_release_notes_contract.sh | 22 ++++++++++--- 4 files changed, 46 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 0e78ccabc..a94d0e927 100644 --- a/README.md +++ b/README.md @@ -794,7 +794,7 @@ internal/cbm/ Vendored tree-sitter grammars (158 languages) + AST extrac Every release is verified through a multi-layer pipeline before publication: -- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, and release notes link every candidate result and the selection evidence. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables. +- **VirusTotal** — all 24 executable candidates (unstripped, debug-stripped and stripped) across the eight release products are scanned before smoke/soak (clean is preferred and only the single-Microsoft `!ml` tolerance documented in [SECURITY.md](SECURITY.md#our-release-policy) may pass; the number of engines that returned a decisive result is recorded as evidence but is VirusTotal fleet availability, not a pass condition). The selected executable is packaged without changing its SHA-256, release notes link the verdict for the exact bytes shipped, and the full per-candidate evidence is published alongside the release as TSVs for anyone auditing the selection. Every distinct object extracted from the shipped containers — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — is then scanned as well, so the full published surface is covered, not just the executables. - **SLSA Level 3** — cryptographic build provenance generated by the trusted GitHub Actions build workflow; verify with `gh attestation verify --repo DeusData/codebase-memory-mcp --signer-workflow DeusData/codebase-memory-mcp/.github/workflows/_build.yml` - **Sigstore cosign** — keyless signatures on all artifacts; bundles included in every release - **SHA-256 checksums** — `checksums.txt` published with every release; verified by both install scripts before extraction diff --git a/SECURITY.md b/SECURITY.md index 678a280ec..48da1d1a1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -126,7 +126,7 @@ Releases are created as **drafts** (invisible to users) and only published after 2. **Sigstore cosign signing** — keyless digital signatures verifiable by anyone 3. **SBOM** — Software Bill of Materials (SPDX) listing all vendored dependencies 4. **SHA-256 checksums** — published with every release -5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link every candidate result and the selection evidence. +5. **VirusTotal scanning** — three behaviourally identical executable candidates — unstripped, debug-stripped and stripped — are derived from one linker output for every release product and scanned before smoke/soak. Zero malicious and zero suspicious is preferred; the only tolerated result is exactly one Microsoft malicious label ending in `!ml`, which is disclosed. The selected executable is packaged without changing its SHA-256. After packaging, every distinct object extracted from the 14 shipped containers is scanned under the same policy — `install.sh`, `install.ps1`, `LICENSE`, `THIRD_PARTY_NOTICES.md`, the MCPB `manifest.json` and the unpacked UI assets — so the covered surface is everything we publish, not only the executables. Release notes link the verdict for the exact bytes shipped; the per-candidate results and the selection table are published as release assets for independent audit. 6. **OpenSSF Scorecard** — repository security health score Scope of the SLSA claim: this is a build provenance claim for release diff --git a/scripts/ci/append-vt-notes.sh b/scripts/ci/append-vt-notes.sh index e9aa62ac5..db4b50c0d 100755 --- a/scripts/ci/append-vt-notes.sh +++ b/scripts/ci/append-vt-notes.sh @@ -211,40 +211,52 @@ asset_base = ( ) -def verdict_cell(target: str, variant: str) -> str: - result = results_by_path[candidate_by_pair[target, variant]["scan_path"]] +# Release notes report the SHIPPED binary and nothing else. +# +# Several candidates per product are scanned so the selector has an alternative +# when an opaque classifier flags one of them, but a reader installing cbm cares +# about the bytes they receive, not about the ones we discarded. The rejected +# candidates' verdicts stay in the published evidence TSVs for anyone auditing +# the selection, and they matter to US in development as a signal; they are +# noise in a changelog. +def shipped_result(target: str) -> dict: + selection = selection_by_target[target] + variant = selection["selected_variant"] + return results_by_path[candidate_by_pair[target, variant]["scan_path"]] + + +def verdict_cell(target: str) -> str: + result = shipped_result(target) label = "clean" if result["policy_classification"] == "clean" else "Microsoft `!ml`" return f"[{label}]({result['virustotal_url']})" -ml_count = sum( - result["policy_classification"] == "microsoft-ml" for result in results -) -engine_counts = [int(result["completed_engines"]) for result in results] +shipped = [shipped_result(target) for target in TARGETS] +ml_count = sum(result["policy_classification"] == "microsoft-ml" for result in shipped) +engine_counts = [int(result["completed_engines"]) for result in shipped] engine_range = str(min(engine_counts)) if min(engine_counts) == max(engine_counts) else f"{min(engine_counts)}–{max(engine_counts)}" section = [ START, "## Security Verification", "", ( - f"Before smoke and soak testing, VirusTotal completed **{len(TARGETS) * len(VARIANTS)} executable scans**: " - "stripped and unstripped candidates for each of the **8 release products**. " - f"Every scan had at least 50 decisive engines (observed range: {engine_range})." + f"Every binary published below was scanned by VirusTotal before smoke and soak " + f"testing, and the verdict for the exact shipped bytes is linked per product " + f"(decisive engines: {engine_range})." ), ( - "All candidates were clean." + "Every shipped binary was clean." if ml_count == 0 - else f"**{ml_count} candidate(s)** had only the documented single Microsoft machine-learning `!ml` result; no other decisive engine reported malicious or suspicious." + else f"**{ml_count} shipped binary/binaries** carried only the documented single Microsoft machine-learning `!ml` result; no other decisive engine reported malicious or suspicious." ), "", - "| Product | Stripped candidate | Unstripped candidate | Shipped |", - "|---|---|---|---|", + "| Product | Shipped binary | VirusTotal verdict |", + "|---|---|---|", ] for target in TARGETS: selection = selection_by_target[target] section.append( - f"| `{target}` | {verdict_cell(target, 'stripped')} | " - f"{verdict_cell(target, 'unstripped')} | `{selection['selected_variant']}` (`{selection['selected_sha256']}`) |" + f"| `{target}` | `{selection['selected_sha256']}` | {verdict_cell(target)} |" ) section.extend( [ diff --git a/tests/test_vt_release_notes_contract.sh b/tests/test_vt_release_notes_contract.sh index 8ced3ceae..6360a2c28 100644 --- a/tests/test_vt_release_notes_contract.sh +++ b/tests/test_vt_release_notes_contract.sh @@ -65,6 +65,12 @@ classes = {(target, variant): "clean" for target in targets for variant in varia classes["linux-amd64", "stripped"] = "microsoft-ml" classes["darwin-arm64", "stripped"] = "microsoft-ml" classes["darwin-arm64", "unstripped"] = "microsoft-ml" +# windows-amd64 draws the tolerated verdict on ALL THREE candidates, so the +# SHIPPED binary carries it and the notes must disclose exactly that one. The +# two targets above keep a clean sibling, so their flagged candidates are +# rejected and must NOT appear in the notes at all. +for _variant in variants: + classes["windows-amd64", _variant] = "microsoft-ml" candidates = [] results = [] by_pair = {} @@ -190,12 +196,18 @@ run_notes "$FIX/current.md" "$FIX/first.md" grep -q 'Intro text.' "$FIX/first.md" || fail "content before section was lost" grep -q 'Outro text.' "$FIX/first.md" || fail "content after section was lost" ! grep -q 'stale data' "$FIX/first.md" || fail "stale section was appended" -grep -q '24 executable scans' "$FIX/first.md" || fail "candidate scan scope missing" -grep -q '8 release products' "$FIX/first.md" || fail "product scope missing" -grep -q '3 candidate(s).*Microsoft' "$FIX/first.md" || fail "tolerated findings not disclosed" +# Release notes report ONLY the shipped bytes. Rejected candidates stay in the +# published evidence TSVs; they are development signal, not changelog content. +grep -q 'verdict for the exact shipped bytes is linked per product' "$FIX/first.md" || \ + fail "shipped-binary scan scope missing" +[ "$(grep -c '^| `' "$FIX/first.md")" = "8" ] || fail "product scope missing: expected one row per release product" +grep -q '1 shipped binary.*Microsoft' "$FIX/first.md" || fail "tolerated finding on a shipped binary not disclosed" +grep -q 'candidate(s)' "$FIX/first.md" && fail "notes must not count rejected candidates" grep -q 'no other decisive engine reported malicious or suspicious' "$FIX/first.md" || fail "non-decisive engines are overstated as clean" ! grep -q 'every other engine was clean' "$FIX/first.md" || fail "non-decisive engines are incorrectly described as clean" -grep -q '| Product | Stripped candidate | Unstripped candidate | Shipped |' "$FIX/first.md" || fail "candidate table missing" +grep -q '| Product | Shipped binary | VirusTotal verdict |' "$FIX/first.md" || fail "shipped-binary table missing" +grep -q 'Stripped candidate' "$FIX/first.md" && \ + fail "release notes must not enumerate rejected candidates" grep -q '| `linux-amd64` |' "$FIX/first.md" || fail "Linux product row missing" grep -q '| `windows-arm64` |' "$FIX/first.md" || fail "Windows product row missing" grep -q 'archive containers were not redundantly submitted' "$FIX/first.md" || fail "no-rescan boundary missing" @@ -228,4 +240,4 @@ if run_notes "$FIX/reversed.md" "$FIX/reversed-capture.md"; then fail "reversed verification markers must fail closed" fi -echo 'PASS: release notes reuse all 24 candidate verdicts, disclose selection, and avoid a duplicate scan' +echo 'PASS: release notes report the shipped binary only, disclose selection, and avoid a duplicate scan'