Optional/QoL: headless-browser verification for web & canvas apps (browser_check) — not roadmap

[itigges22]

Status: exploratory, purely QoL — NOT on the roadmap, not committed to building. Capturing the idea + research so it isn't lost.

The gap

The agent can verify Python (run it, catch crashes) and a web server (curl → 200 + HTML), but has no feedback loop for what actually runs in the browser — uncaught JS exceptions, canvas rendering, gameplay logic. So it can ship a page that "serves 200" yet is broken in a browser (the recurring example: a generated Snake game that renders but doesn't move).

The idea

A browser_check tool: load the served page (http://localhost:PORT) in a headless browser and report uncaught JS exceptions + console errors; optionally simulate input (keypress/click) and screenshot for a later vision check. (A working proxy tool + sandbox endpoint were prototyped and then dropped — see git history if revisiting.)

Why it's not being built now

• It would not fix the main pain (logic/visual correctness like the Snake dx/dy bug — no JS error, just wrong). That class needs a vision loop (screenshot → multimodal model), which is gated on running a vision-capable model. A console-error rail catches a narrower, different class (JS crashes).
• The bigger quality lever already shipped: the lens-training loop (/good·/bad + atlas lens retrain) catches bad writes more generally.

Research findings (adversarially verified — so a future attempt doesn't re-litigate)

• jsdom / happy-dom: unreliable even for error capture — async exceptions escape to Node and can crash the host; no real canvas. Not viable.
• Lightpanda: AGPL-3.0 (license), no rendering engine (no canvas/screenshots), Beta; headline RAM/speed claims didn't survive verification. Not viable for this.
• Servo / servoshell: genuinely rasterizes canvas, but too immature for headless container automation as of 2026 (feature-flagged, screenshot/automation maturity unestablished).
• Viable path = real Chromium headless-shell (Playwright --only-shell, or @sparticuz/chromium ~130 MiB uncompressed, or the chromedp/headless-shell image): real engine, does both tiers, ~150 MB not ~1 GB. Should be an opt-in service (compose profile) so the default sandbox stays lean.

If ever revisited

Build it once, properly: headless-shell + input simulation + screenshot + vision, as an opt-in atlas-browser service — when there's a vision-capable model to make the visual/logic check real. Don't ship a partial Tier-1-only rail that gets rebuilt later.

[itigges22]

Design update: text-native behavioral verification (no vision dependency). A 2026-07-18 dogfood session made the gap concrete: python compiled, Flask booted, GET / returned 200, V3's sandbox verified every file, and the shipped snake game was frozen. Three defects, zero errors anywhere: a game-over check keyed on input state instead of self-collision, an inverted grow-on-food branch, and a static/game.js written but referenced by nothing. Everything we verify today is server-side; nothing executes the JS against a DOM.

The plan is three layers, each returning plain text into the same repair channel sandbox stderr uses now. The model never needs to see an image: where screenshots are involved, the harness compares them and reports booleans, so any GGUF works (model-agnostic stays the rule).

1. Structural floor (landed on dev, 7526224). Session file manifest: later files are generated knowing what earlier files exist, and the session's written files are merged into V3's project_context (which previously carried only files the model had read, so written-but-never-read siblings were invisible to candidate generation). Plus an asset-graph lint: templates no render_template names, static/ files nothing references, src/href/url_for targets that don't exist. Advisory notes, deduped, never blocking; the stuck-pattern detectors remain the only loop-breakers.

2. Runtime telemetry probe (this issue's minimal version). Headless Chromium in the sandbox image. Load the page, dispatch synthetic input, emit a text report: console errors verbatim; dialog hooks ("alert('Game Over') fired 80ms after first keydown"); loop liveness (is a rAF/setTimeout loop still scheduling after the interaction); change booleans (canvas hash changed or not over the probe window, DOM mutation count, listener registry). Four lines of that report fully localize the frozen-snake bug for a text-only model.

3. Model-authored DOM assertions (the semantic layer). Extend the existing self-test generation to the DOM: at write time the model emits a small assertion script against its own variable names ("dispatch ArrowRight, wait 300ms, assert head.x increased"); the sandbox runs it and returns pass/fail with observed values. Intent-vs-behavior diffs are exactly what the repair loop already consumes for Python.

Chromium in the sandbox image (~300MB) is the one heavy dependency, and it belongs to layers 2 and 3 only; layer 1 costs nothing at runtime. Keeping this issue as the tracker for layers 2 and 3.