chore(ci): prove and prevent intermittent GitHub Actions runner-assignment stalls

[seonghobae]

Problem

Noema has historical evidence of required GitHub Actions runs remaining queued without runner assignment or logs, but current evidence does not support a repository-wide hosted-runner outage. Protected main integrated #76 at c85d710804139c0697d7ef8fa47d02b1389e6d84, and multiple post-integration PR heads receive GitHub-hosted runners normally.

The remaining repository-owned work is to integrate a bounded runner-assignment diagnostic without regressing the clean dependency/package-manager control plane, exercise it from protected source, and keep any historical organization/platform cause unresolved unless authorized evidence proves one.

Fresh current evidence

• PR fix(governance): restack independent approval audit after nanoid #90 head 4ee417372c7a7e084767b50692c376bf03ff4b0a: application CI, reviewer-ci and central Security Scan terminal-success.
• PR build(reproducibility): restack deterministic npm toolchain #91 head ebde3720ee27292be0ce3315986e0438dd66af9f: application CI, reviewer-ci and central Security Scan terminal-success.
• The runner-assignment implementation has been rebuilt on build(reproducibility): restack deterministic npm toolchain #91 as Draft feat(operations): restack runner-assignment audit on deterministic toolchain #94. Its earlier exact head 4b3a6d7896ae1c28de02aecd2542b62b0dcd1937 completed application CI 31378206257 and reviewer-ci 31378206300 successfully. Central Security Scan is absent on the feature-base stack and is therefore non-passing defer_until_trigger, not scanner success.
• Direct-main Draft feat(operations): audit GitHub Actions runner assignment #88 was closed as superseded only after feat(operations): restack runner-assignment audit on deterministic toolchain #94 preserved its full 10-path workstream and produced fresh application/reviewer evidence. Old feat(operations): audit GitHub Actions runner assignment #88 checks/reviews do not transfer.

Therefore the following hypotheses are falsified for the observed current repository state:

• Actions globally disabled for Noema;
• GitHub-hosted ubuntu-latest universally unavailable;
• every current workflow stuck before assignment.

These observations do not identify the cause of historical stalls.

Repository-owned diagnostic lineage

Draft #94 (feat/actions-runner-assignment-audit-on-toolchain) is the current implementation owner. It preserves #91's deterministic Node/npm/install-script/lockfile controls and adds the read-only operations:runner-assignment evidence plane.

The evaluator separates:

• PASS: runner assignment evidence only;
• PENDING: no isolated runner-allocation conclusion, including environment/dependency waits; nonzero and non-passing;
• FAIL / runner_assignment_stalled: only when a queued run+job has no assignment evidence anywhere in that selected run beyond bounded grace, or evidence is malformed/stale.

The operator reads fully paginated filter=all workflow-job evidence and binds selected runs to one exact pull-request source head. It cannot satisfy a required Check, formal review, merge, release or deployment gate.

The current successor also narrows the gh child-process environment: only PATH, the read-only GH_TOKEN, pinned GH_HOST=github.com, and NO_COLOR=1 cross the subprocess boundary. Ambient GITHUB_TOKEN, NVIDIA credentials, Maintainer/Reviewer App material, proxy variables and HOME-scoped credentials are excluded rather than inherited through process.env.

#91 is not a causal dependency of runner assignment; it is the source-control baseline that must not be regressed. Safe integration order is:

#91 protected integration
→ rebuild/refresh #94's bounded runner-assignment delta onto the then-current protected lineage
→ fresh application CI/reviewer/Security Scan/review evidence
→ protected-main operational runner-assignment exercise

Remaining historical RCA questions

Repository-visible evidence cannot establish:

• organization/enterprise Actions enablement or policy history;
• billing/spending/usage holds where applicable;
• hosted-runner or runner-group access history;
• organization-wide concurrency/queue saturation at historical timestamps;
• enterprise IP allow-list/policy effects;
• GitHub platform incident/provider contribution at the affected timestamps.

Do not infer any of these from a queued state alone.

Acceptance criteria

Repository-owned controls

• Preserve required CI/reviewer/security workflows; no synthetic replacement status.
• Demonstrate current workflows receive hosted runners on multiple independent exact heads.
• Keep environment/dependency waits distinct from isolated runner-assignment stalls.
• Preserve a bounded true-stall classification for queued run+job with no assignment evidence after grace.
• Integrate fix(security): update transitive nanoid for CVE-2026-67213 #76 so the inherited nanoid blocker is removed from protected main.
• Replace stale direct-main feat(operations): audit GitHub Actions runner assignment #88 with dependency-ordered feat(operations): restack runner-assignment audit on deterministic toolchain #94 while preserving the full runner-assignment workstream.
• Prevent the diagnostic gh subprocess from inheriting unrelated model/repository/App/proxy credentials.
• Integrate build(reproducibility): restack deterministic npm toolchain #91 or an equivalent reviewed deterministic package-manager baseline.
• Refresh feat(operations): restack runner-assignment audit on deterministic toolchain #94 onto the integrated protected lineage and reacquire fresh exact-source/current-base application CI, reviewer-ci, eligible Security Scan and current review evidence.
• From protected main, exercise the assignment audit against at least one assigned/running workflow and controlled malformed/stale evidence.
• If an unassigned stall recurs, retain exact head/run/full paginated jobs/timestamps/assignment evidence before source changes.
• Keep assignment evidence separate from check conclusion, review, approval and merge authority.

Organization/platform RCA

• Obtain authorized Actions enablement/hosted-runner policy history.
• Check billing/usage restrictions where applicable.
• Check runner-group/access and organization concurrency for the affected period.
• Check enterprise policy/IP restrictions where applicable.
• Compare affected timestamps with platform incident evidence.
• Identify a falsifiable historical cause or explicitly retain unresolved status if evidence remains insufficient.

Guardrails

Do not merge with queued/pending/failed required checks; do not change runs-on merely to hide an unproven organization configuration issue; do not remove reviewer/security/coverage/dependency gates; do not create no-op replacement statuses; do not label environment/dependency waits as runner allocation failures without isolated evidence.

Related: #27, #29, #77, #79, #91, #94

[seonghobae]

2026-08-03 follow-up

The latest PR #28 head is edc693bc5cf3e43d010c528d8438a34ca672d10e. GitHub reports the PR as mergeable and CodeRabbit is successful, but all current-head GitHub-hosted jobs are still waiting for runner assignment:

• ci run 30795894312 → job 91629278475 (verify) remains queued; no execution steps or logs
• reviewer-ci run 30795894333 → job 91629278294 (reviewer) remains queued; no execution steps or logs
• Security Scan run 30795894282 → jobs 91629281319 (osv-scan), 91629281336 (dependency-review), 91629281381 (trivy-fs), and 91629281469 (scorecard) remain queued; only cancel-closed-pr-runs is skipped

GitHub's public status page currently reports Actions as operational and does not show an active Actions incident for this period. That does not prove the repository configuration root cause, but it makes organization/repository billing, Actions enablement, hosted-runner policy, runner-group access, concurrency, or enterprise restrictions the next checks rather than weakening workflow definitions.

The PR will remain unmerged until these exact current-head jobs start and reach successful terminal conclusions. Repeated commits have reproduced the same queue condition, so no further no-op retrigger commits should be used as remediation.

[seonghobae]

2026-08-03 runner recovery and current-head state

Runner assignment partially recovered on PR #28 head 94c7d93a56bf74a5e855d22cd4ba4d19fc1128a3:

• reviewer-ci run 30796803567 completed successfully.
• Security Scan run 30796803596 completed successfully.
• ci run 30796803509 started and produced logs; it failed on three malformed Vitest parameter rows rather than on runner infrastructure.

The test table was corrected in ee776fc2f0f27abffb9a91596da7ea4ee5e7f2bc, the feature branch was fast-forwarded through the current main merge result so it is no longer behind, and the changelog trace was added. Current PR head is b77263b5c7b127ba2e266141c1315f16ec2ec0b3.

Current-head runs are again queued without runner assignment at this inspection:

• ci run 30798515233
• reviewer-ci run 30798515122
• Security Scan run 30798515064

This narrows the incident from a permanent repository disablement to intermittent or delayed GitHub-hosted runner assignment across independent workflows. The merge remains blocked until these exact current-head jobs execute and reach successful terminal conclusions. No workflow or gate has been weakened.

[seonghobae]

2026-08-03 recovery evidence

PR #28 head b77263b5c7b127ba2e266141c1315f16ec2ec0b3 received runner assignment and all current-head workflows completed successfully:

• ci run 30798515233 / job verify: success, including npm run release:verify.
• reviewer-ci run 30798515122 / job reviewer: success, including the 100% line+branch coverage and 100% docstring gates.
• Security Scan run 30798515064: scorecard, osv-scan, trivy-fs, and dependency-review all succeeded.
• CodeRabbit current-head status: success.
• Unresolved review threads: 0.

PR #28 was then SHA-bound squash-merged as 82feeb3200d4e0c318545d1f896f6e00784d229e.

The immediate merge blocker has recovered, but this issue remains open because the underlying organization-level runner-assignment cause has not been established and the acceptance criteria require reliable recurrence evidence rather than one recovered head. Do not weaken or remove the workflows.

[seonghobae]

Additional recurrence evidence

PR #31 head 2fc98d606e7e9ae5287cb04c960ff09b3c683c2b also received normal GitHub-hosted runner assignment:

• ci run 30803093531: success (release verify succeeded).
• reviewer-ci run 30803093522: success (100% line+branch and docstring gates succeeded).
• Security Scan run 30803093570: scorecard, osv-scan, trivy-fs, and dependency-review all succeeded.

PR #31 was SHA-bound squash-merged as e7e6dbac826a77260a32ec3df3285fe7892f3885. This is a second consecutive recovered head, but the original organization-level queuing root cause remains undocumented, so the issue remains open pending configuration/billing/concurrency evidence.

[seonghobae]

Additional current-head evidence (2026-08-04): PR #59 head 677180bc5ff599ae579784eea4bb6dca2a688060 created the three required pull-request workflows, but all remain queued without runner assignment or logs at inspection time:

• ci run 30876945749, job 91890238070 (verify): queued
• reviewer-ci run 30876945759, job 91890238015 (reviewer): queued
• Security Scan run 30876945723: scorecard job 91890238333, dependency-review 91890238353, osv-scan 91890238390, and trivy-fs 91890238404 queued

The PR is not being merged and no required context is being bypassed. This reproduces the cross-workflow runner-assignment condition on a new exact head after the earlier recoveries.

[seonghobae]

2026-08-06 additional runner-assignment evidence

PR #65 exact head 9e03b17d0bdda75809f394aaafb0b44cf1267a51 received normal GitHub-hosted runner assignment across all required workflows:

• ci run 31031918931: completed successfully.
• reviewer-ci run 31031918958: completed successfully, including 331 tests and the documented 100% production statement, branch, and public-docstring gates.
• Security Scan run 31031918997: completed successfully.

This is further evidence that the earlier cross-workflow queue condition is not currently reproducing. The issue remains open because the original organization-level cause and configuration/billing/concurrency evidence are still not established; no workflow, runner label, or required gate was weakened.

[seonghobae]

2026-08-06 runner-assignment recovery evidence

PR #65 exact head 44e3cb839a9769e6ad795e4e16f4015b1548f506 received normal GitHub-hosted runner assignment and terminal results for every current-head workflow:

• ci run 31065763303, job 92503029261: completed successfully; release verification executed.
• reviewer-ci run 31065763329, job 92503029119: completed successfully; 332 tests passed, production statement/branch coverage remained 100%, and public docstring coverage remained 100%.
• Security Scan run 31065763285: dependency-review job 92503029324, OSV job 92503029351, Trivy job 92503029360, and Scorecard job 92503029399 all completed successfully. Conditional fallback/report steps were skipped only where their primary path succeeded; the closed-PR cancellation job was expectedly skipped because PR feat(sandbox): add quarantined patch validation #65 is open.

This demonstrates current recovery for one integrated exact head and no missing job logs. It does not establish the original organization-level root cause or long-term reliability, so #30 should remain open until Actions/billing/runner-group/concurrency policy is audited and the remediation/root-cause evidence required by the acceptance criteria is attached. No workflow or required gate was weakened.

[seonghobae]

2026-08-07 recurrence evidence: PR event has no observable Actions runs

PR #69 exact head 9abe3be2220361cb7cb289325f3f03e6e26bf739 is open against main, but the commit-scoped Actions query currently returns no pull-request-triggered workflow runs at all for this exact head. This differs from the earlier queued-run symptom: there are no current-head ci, reviewer-ci, or Security Scan run objects to inspect, so there is nothing safe to rerun and no terminal check evidence to treat as success.

The repository's ci workflow remains configured for pull_request; no workflow gate is being weakened or replaced. PR #69 therefore remains blocked on missing exact-head CI/security/coverage evidence. No no-op retrigger commit or repair workflow will be used. The current PR body records the missing-run state explicitly.

This comment is evidence only; it does not establish whether the cause is Actions event suppression, repository/organization policy, App/token event semantics, or runner infrastructure. #30 should remain open until the relevant Actions/billing/policy/event-delivery configuration can be audited with authoritative evidence.

[seonghobae]

2026-08-07 runner/event recovery evidence

PR #69 has since advanced to exact head eaf712097d11a830667971095f9dcf985cc2978f. Commit-scoped pull-request workflow discovery now returns terminal runs instead of the prior no-run state:

• ci run 31136829151 / verify: completed successfully; release:verify executed, 68 test files / 764 tests passed, production statement/branch/function/line coverage were 100%, and npm reported zero vulnerabilities.
• reviewer-ci run 31136829154 / reviewer: completed successfully, including its 100% line+branch coverage and 100% public-docstring gates.
• Security Scan run 31136829152: Scorecard, dependency-review, Trivy filesystem, and OSV jobs all reached successful terminal conclusions.

This documents recovery from the missing-run recurrence but does not establish the original Actions/event-delivery/runner-assignment root cause or satisfy the configuration/billing/concurrency audit acceptance criteria. #30 therefore remains open. No runner label, required check, workflow gate, or protection has been weakened.

[seonghobae]

2026-08-09 fresh queue evidence: current PR #71 exact head 768882e173de23f521f6ca40534e0ee390c85c8b has three independently created PR workflow runs whose executable jobs have not received a runner: ci run 31310403045 → job 93237244647 (verify) is queued with no steps/logs; reviewer-ci run 31310403046 → job 93237244482 (reviewer) is queued with no steps/logs; central Security Scan run 31310403060 → scorecard, dependency-review, trivy-fs, and osv-scan jobs are all queued with no steps/logs (only the unrelated cancel-closed-pr-runs condition is terminal-skipped). Separately, PR #67's targeted patch-validator-image retry is also still queued, while older #76 exact-head runs completed successfully. This supports an intermittent/shared runner-assignment or organization capacity/policy boundary rather than a #71 source failure: the queued jobs have not begun source checkout. Repository source mutation or repeated commits cannot repair this boundary. Acceptance still requires checking organization/repository Actions enablement, billing/spending/usage, hosted-runner/runnerset access and queue/concurrency state as this issue lists. Pending/queued evidence remains non-passing; no required workflow was weakened or bypassed.

[seonghobae]

Additional current feasibility evidence: GitHub's official status page presently reports Actions as operational and no current platform-wide incident (https://www.githubstatus.com/). Combined with #71's independently queued ci, reviewer-ci, and Security Scan jobs having no runner-assigned steps/logs, while older #76 jobs completed, this makes a current global GitHub Actions outage a weaker hypothesis. The highest-value remaining probes are organization/repository-specific Actions enablement, hosted-runner eligibility, billing/spending/usage hold, runner-group access, and shared concurrency/queue saturation. This remains an inference, not a proven root cause, until those control-plane settings are inspected.