Fix Real World Live Test + stop streaming JSON to the terminal - #80
Merged
Conversation
… timeout The Nightly Real World End To End Test has failed every night since 07-17. It is not the mocks (this test clones live prod repos) and not the scraper data (the sibling test-nightly.yml reads the same live data and stays green). The govbot source is unchanged since the #71 rename, so the trigger is external. Root cause: the pipeline's tag step downloads an 87MB all-MiniLM-L6-v2 ONNX model from HuggingFace into a fresh /tmp dir on every run. HF rate-limits anonymous downloads per source IP, and GitHub Actions shares a pool of egress IPs, so the anonymous quota is collectively exhausted -- the model download gets throttled to a crawl (~87MB over >10min) and blows past the tape's VHS wait window. The pipeline actually completes ("Pipeline complete!" in logs); VHS just times out first because rc:0 never renders while stdout sits idle during the long download. A prior attempt to simply raise the timeout (300s->600s) still failed at ~23min: the throttled download exceeds even 600s. Fix: cache the model + tokenizer across runs (actions/cache), downloading once with curl --retry on a cache miss, and seed them into the test dir before each VHS attempt. govbot skips the download when model.onnx + tokenizer.json already exist, so the timed pipeline no longer re-downloads. Tape wait stays at 300s; job timeout kept at a modest 20min to absorb the occasional cache-miss download. Separate scraper data bug (malformed from_organization) tracked in #78. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NFovdWwJP6CdigoR2imsJy
sartaj
force-pushed
the
fix-synthetic-test-timeout
branch
from
July 21, 2026 03:38
fa1f044 to
9f8658c
Compare
…yd wedge)
Caching the model (previous commit) removed the slow HF download, but the test
still failed: the pipeline completed on disk ("Pipeline complete!", docs/ written)
yet VHS timed out because the rc:0 completion sentinel never rendered. The screen
froze right after the tag step streamed a bill's raw JSON to stdout.
Root cause: the tape kept the pipeline's stdout (raw bill JSON) on screen. A burst
of it wedges the ttyd terminal so subsequent output — including the rc:0 sentinel
the tape waits on — never renders. This surfaced when scraper data changed what
streams through tagging, which is why it began failing without any govbot change.
govbot itself exits cleanly (verified locally under a PTY with a fresh clone).
Fix: redirect the pipeline's stdout to a file too (stderr was already redirected
to dodge a clone-progress PTY deadlock), run it in the background, and print a '.'
heartbeat every 2s so the VHS screen keeps updating without any JSON on it, then
echo a clean rc:0 sentinel. Verified locally: screen shows "..rc:0", JSON lands in
/tmp/pipeline.out, and docs/feed.xml is produced.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NFovdWwJP6CdigoR2imsJy
sartaj
changed the title
sartaj
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Nightly Real World End To End Test (
synthetic-test.yml) has failed every night since 2026-07-17 (last pass 07-16).Diagnosis
gu,wy) into/tmp; the checked-in mocks only back the Rust snapshot tests.test-nightly.ymlclones + reads the same live data every night and has stayed green through 07-20. If the data were the problem, it would fail too.src/is byte-identical between the 07-16 pass and 07-17 fail (no commits since the Rename .govbot data catalog to govbot_data (#22) #71 rename on 07-15). So the trigger is external to the repo.Root cause: the
tagstep downloads the 87 MBall-MiniLM-L6-v2ONNX model from HuggingFace into the fresh/tmp/govbot-test/govbot_data/on every run (no cache). The URL is unpinned (.../resolve/main/...) and now redirects to HF's newer Xet CDN (us.aws.cdn.hf.co/xet-bridge-us/...), which throttles anonymous cloud-IP (GitHub runner) downloads. That pushes the pipeline just past the tape'sWait+Screen@300s /rc:0/window, so VHS times out — even though the pipeline does finish (pipeline.logshows "Pipeline complete!"). Locally the whole pipeline runs in ~6 s.Fix
Keep the real-world download — it's the whole point of this test (a fresh user hits the same download). Just give it headroom:
Wait+Screen@300s→@600stimeout-minutes: 15→25(2× retries at 600 s would otherwise blow the 15-min job cap)A passing run exits
Wait+Screenthe instantrc:0appears, so the larger ceiling only costs wall-clock on a genuinely slow/failing attempt.Verification
Both edited files match this workflow's
pull_request.pathsfilter, so this PR run itself is the test — a green run confirms the download-vs-timeout diagnosis. If it's still red, the throttled download exceeds even 600 s, pointing to a product-side downloader fix (retry/resume, HF token, or pinned mirror) rather than a bigger timeout.Not in scope
A separate, real scraper data regression found during investigation —
from_organization/organization_idemitted as unresolved"~{...}"references across all scraped repos — is tracked in #78 (originates upstream inchn-openstates-scrape, not govbot; does not affect this test).🤖 Generated with Claude Code