CI: smoke tests for scanner + dashboard render paths#29
Merged
Conversation
Closes the second open-source-readiness blocker — no CI beyond
deploy. Up until now a broken import or a runtime throw in a
render function could sail onto main and break production, as
happened on 2026-04-18 with older manifests missing aiSystems.
## What's here
- `.github/workflows/ci.yml` — runs on every push and PR, in
its own concurrency group so superseded runs get cancelled.
Two steps:
- Scanner smoke: runs `npm run scan -- .` against this repo.
Exercises every scan rule, policy render, framework
evaluation, and report generator end-to-end on a real
tree. Non-zero exit fails the job.
- Dashboard render smoke: runs scripts/smoke-dashboard.ts
via tsx. Exercises every exported render function with
both a current-shape and a pre-Phase-8-shape manifest.
- `scripts/smoke-dashboard.ts` — builds two fixtures (new-shape
and old-shape), summarises each, and feeds them through
renderDashboard, renderRepoDetail, renderNistView,
renderBranchComparison, renderTrendChart,
renderAIComplianceView, and renderInventoryView. Also
independently exercises evaluateEUAIAct /
calcAIComplianceScore / getAIPhaseScores.
Assertions per render: returns a string, at least 100 bytes,
no bare `>undefined<` or `="undefined"` anywhere. Inline JS
references to `undefined` as an identifier are allowed.
The old-shape fixture specifically lacks aiSystems — mirrors
what pre-Phase-8 scanner versions stored in KV and guards
against the 2026-04-18 regression.
## Not here
tsc --noEmit is intentionally skipped: this project uses tsx
at runtime, which tolerates the `verbatimModuleSyntax` +
no-@types/node combination that `tsc` does not. Running `tsc
--noEmit` against the current tsconfig produces hundreds of
pre-existing errors unrelated to real defects. A proper lint
pass (Biome, or tsconfig cleanup + ESLint) is tracked as a
separate open-source-readiness item.
## Checklist
- Closed: "No CI beyond deploy"
- Still open: "No unit tests" (smoke tests are a down payment,
not a replacement) and "No lint / type-check"
This was referenced Apr 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second open-source-readiness blocker: no CI beyond deploy. Closes that gap with a pair of smoke tests that run on every push and PR.
Why smoke tests, not tsc --noEmit
tsc --noEmitagainst the current tsconfig produces hundreds of pre-existing errors unrelated to real defects — the project usestsxat runtime, which tolerates theverbatimModuleSyntax+ no-@types/nodecombination thattscrejects. Fixing that cleanly is a separate effort. More importantly, a type-check would not have caught the 2026-04-18 outage (the offending field was optional and legallyundefined). Runtime smoke tests with fixture manifests would have.What's in the workflow
.github/workflows/ci.yml— runs on push to main and every PR. Own concurrency group withcancel-in-progress: trueso force-pushes don't waste minutes. Two steps afternpm ci:Scanner smoke —
npm run scan -- .against this repo. Exercises every scan rule, policy render, framework evaluation, and report generator on a real tree. Catches any regression that breaks the scanner pipeline end-to-end.Dashboard render smoke —
npx tsx scripts/smoke-dashboard.ts. Exercises every exported render function with two fixtures: current-shape and pre-Phase-8-shape (noaiSystemsfield). The second fixture is a regression guard for the 2026-04-18 outage.scripts/smoke-dashboard.ts— constructs both fixtures, summarises each, feeds throughrenderDashboard,renderRepoDetail,renderNistView,renderBranchComparison,renderTrendChart,renderAIComplianceView, andrenderInventoryView(empty + populated). Also independently exercisesevaluateEUAIAct/calcAIComplianceScore/getAIPhaseScores. Assertions per render: returns a string, at least 100 bytes, no bare>undefined<or="undefined"in output. Inline JS that referencesundefinedas an identifier is allowed.Test plan
npx tsx scripts/smoke-dashboard.tspasses locally.npm run scan -- .exits 0 locally.Known gaps (tracked in checklist)