fix(queue): prune the failed_jobs table on a schedule - #138
Merged
Conversation
failed_jobs had never been pruned and had grown to 1916 rows on production, the oldest from 2026-04. Every row is a resolved incident, and the volume drowns the handful that still matter — `queue:failed` shows only the tail, so the table reads as "3 old entries" when it is nearly two thousand. `queue:prune-failed --hours=720` keeps 30 days, which is long enough to investigate a failure after the fact. Verified the schedule entry registers (`15 4 * * *`, next due 19h). Found while validating the current failures rather than assuming they were stale: of the 17 since 2026-07-24, all but one are DistillTeamEventsJob at 01:31 nightly, which stopped after 2026-07-29 (fix #131, 2026-08-01 — six clean nights since, with `memory:distill-events` confirmed still scheduled and due, so the silence is a fix and not a job that quietly stopped running). The remaining one is the browser_task max_steps 400, fixed in #136.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finding
failed_jobson production holds 1916 rows, oldest from 2026-04, and had never been pruned — there is noqueue:prune-failedanywhere in the scheduler.This is not cosmetic.
queue:failedprints the tail, so the table reads as "a few old entries from April" when it is actually nearly two thousand rows spanning four months. That is exactly how the recent, still-relevant failures get missed.Change
30 days is long enough to investigate a failure after the fact. Verified the entry registers on the container:
15 4 * * * php artisan queue:prune-failed --hours=720 — Next Due: 19 hours. The first run prunes the backlog; no manual deletion of production rows needed.What the failures actually were (validated, not assumed)
17 failures since 2026-07-24:
DistillTeamEventsJob(AiAccessUnavailableException+ a 401x-api-key header is required)RunPlanningStage(AI access, then MaxAttempts)BuildArtifactJob(tool call validation failed … /max_steps)The distill job ran nightly at 01:31 and failed every night from 2026-06-25 to 2026-07-29, then stopped.
"Stopped failing" was checked against "stopped running":
memory:distill-eventsis still registered (30 1 * * *, next due in 16h), so the six clean nights since are the fix working, not a job that quietly fell out of the schedule.Older classes in the table follow the same shape — each stops at the date its fix shipped (e.g. 39 ×
ExperimentTransitioned | BroadcastExceptionending 2026-06-29).Verification
pint + phpstan clean; schedule entry confirmed live on the production container.