fix(calm-suite): validate remediated CALM before opening a CALMGuard PR#2693
Draft
eddie-knight wants to merge 1 commit into
Draft
Conversation
CALMGuard's remediation flow opened a GitHub PR with the remediated CALM document without validating it first — so a malformed change (most realistically an LLM-supplied control key that violates the schema's ^[a-zA-Z0-9-]+$ pattern), or any future regression in applyChangesToCalm, could be committed and PR'd as a schema-invalid document, undermining cross-tool interop. - Validate the remediated output via the existing validateWithCalmCli (the same canonical CALM CLI check the /api/calm/validate route uses) right after applyChangesToCalm and BEFORE any GitHub mutation. On failure the stream emits an error and aborts — no branch, commit, PR, or label is created (fail-closed). - Pin the route to the Node runtime (export const runtime = 'nodejs'); the validator shells out via child_process, which is unavailable on Edge. Matches /api/calm/validate. - Distinguish a genuine schema-invalid document from a validator that failed to run (clear "could not validate" message), both fail-closed. Scoped to remediation write-backs (pipeline/infra commit YAML/IaC, not CALM). The input document is Zod-checked at ingest; this gates the committed output. Adds a route-level test: an invalid result never reaches githubFetch (no commit), a valid result proceeds past the gate. Signed-off-by: Eddie Knight <knight@linux.com>
Contributor
Author
|
Stacked on #2692, can be taken out of draft when that's merged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The journey this fixes: You run a CALMGuard analysis, then click "Create remediation PR" to have CALMGuard open a pull request that applies its compliance fixes to your CALM document. Today, CALMGuard commits and opens that PR without validating the remediated document first — so if the remediation produces a schema-invalid CALM file, an invalid document lands in a PR against your repo, and downstream tools (the CLI validator, the Hub, anything consuming CALM) choke on it.
The most realistic way this happens:
applyChangesToCalmwrites an LLM-supplied control key (change.after) directly as a control-object key, with no check against the CALM schema's key pattern^[a-zA-Z0-9-]+$. An LLM emitting something like"PCI DSS Req 8.4.2"as a key yields a schema-invalid document — and nothing was stopping it from being committed.After this PR, the remediated document is validated before any GitHub write:
/api/calm/validateroute uses) right after the merge and before the branch/commit/PR.What changed
src/app/api/github/create-pr/route.ts(remediation branch): afterapplyChangesToCalmproduces the remediated CALM, runvalidateWithCalmCli; on{valid:false}or a validator throw,throw→ the route's existing SSE error/close path aborts before the first GitHub mutation (createBranch).export const runtime = 'nodejs') — the validator shells out viachild_process, which is unavailable on the Edge runtime. Matches/api/calm/validate.src/__tests__/api/create-pr.test.ts): an invalid result never reachesgithubFetch(no commit); a valid result proceeds past the gate.Scope notes
pipeline/infrawrite-backs commit YAML/IaC, not CALM, so they're untouched.Type of Change
Affected Components
(All unchecked — surface is CALMGuard (
calm-suite/calm-guard/), which has no template checkbox.)Commit Message Format ✅
Conventional Commits, DCO signed-off:
fix(calm-suite): validate remediated CALM before opening a CALMGuard PRTesting
Verified on Node 22:
npm run typecheck --workspace=calmguard(clean),npm run lint --workspace=calmguard(clean),npm run test:run --workspace=calmguard(119 pass, incl. 2 new), andnpm run build --workspace=calmguard. The new route-level test mocks the CLI validator, the LLM remediation agent, and the GitHub client, and asserts that an invalid validation result aborts the stream and never callsgithubFetch(no GitHub write), while a valid result proceeds past the gate.Checklist