Skip to content

fix(runtime): atomically finalize Run projections and repair missing terminal events #416

Description

@AsperforMias

Area

Runtime

Summary

Severity: P1. Terminal Run persistence is split across independent D1
writes. If the Run status update commits but the Session lifecycle projection
does not, the Run becomes terminal while the owning Session remains RUNNING.
The stale-run reconciler only considers active Runs, so it does not repair this
state and the next Run admission is rejected.

The same terminal-finalization boundary persists the canonical terminal
session_event separately. A Driver replay can repair that event while the
Driver remains alive, but there is no durable repair obligation or background
scanner for a terminal Run that lacks its terminal event after the Driver or
Sandbox has also been lost.

Steps to reproduce

  1. Check out main at b4a5f674bcc05ad5ba5221ea1d82c95dbc20e6b9.
  2. Create the local SQLite D1 Session/Run fixture with one running Run and a
    RUNNING Session.
  3. Wrap D1 so the session_run terminal CAS succeeds, then the immediately
    following UPDATE session fails.
  4. Call the real terminal Run status transition with status: completed.
  5. Query the Run and Session, invoke the real stale-run reconciliation, then
    attempt another Run admission.
  6. Separately inject a failure for the terminal session_event insert after
    terminal Run persistence and do not replay the Driver event.

Expected behavior

Terminalization must either atomically persist the Run state, Session lifecycle
projection, and canonical terminal event, or leave a durable, idempotent repair
obligation that scheduled maintenance can complete without a live Driver.

A terminal Run must never leave its Session unable to accept the next Run, and
terminal event visibility must not depend on a surviving Driver replay.

Actual behavior

The first fault injection produced:

{
  "run_status": "completed",
  "session_status": "RUNNING",
  "staleReconciliation": false,
  "nextAdmission": "Session cannot accept a new run."
}

Repeating the same terminal transition can repair the Session projection, but a
process/Driver loss before that repeat leaves the Session stranded. For the
second injection, the terminal event can be restored only when the Driver
replays the event; no scheduled scan finds and repairs the missing terminal
receipt.

Evidence

  • setSessionRunStatus() updates session_run before the separate Session
    lifecycle projection in
    apps/api/src/modules/runtime/infrastructure/session-runs/session-run-write.repository.ts.

  • Stale reconciliation scans active Run states, so a completed Run is not a
    repair candidate:
    apps/api/src/modules/runtime/application/session-runs/stale-run-reconciliation.service.ts.

  • Terminal runtime event persistence is separately fault-injected by the
    existing regression test:

    bun test apps/api/tests/runtime-final-output-ingestion.test.ts --test-name-pattern 'preserves a long final snapshot across hibernation, terminal failure, and replay'
    

    The test passes only after an explicit replay; it does not prove a
    Driver-independent repair.

Related work is deliberately not a duplicate: #329 fixes initial Session Run
admission atomically; #78, #83, and #90 handle interruption presentation,
tool-result display, or stale active Runs. None makes terminal D1 projections
atomic or repairable after a lost Driver.

Environment

  • mosoo commit: b4a5f674bcc05ad5ba5221ea1d82c95dbc20e6b9
  • Local Bun 1.4.0-canary.1 and SQLite-backed D1 fixture
  • Fault injection at the D1 boundary; no Cloudflare account, Sandbox,
    credential, provider request, or billable resource

Contribution

I can test a fix.

Checklist

  • I searched existing issues before opening this report.
  • I included a reproducible path.
  • I removed or redacted secrets from logs, screenshots, and config.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions