Skip to content

fix(sentry): drop Redis connect-time failures from the container-restart race - #134

Merged
escapeboy merged 1 commit into
developfrom
fix/sentry-filter-redis-connect-race
Aug 3, 2026
Merged

fix(sentry): drop Redis connect-time failures from the container-restart race#134
escapeboy merged 1 commit into
developfrom
fix/sentry-filter-redis-connect-race

Conversation

@escapeboy

Copy link
Copy Markdown
Owner

Fixes the noise behind Sentry issue #957 (Predis\Connection\Resource\Exception\StreamInitException: Connection refused [tcp://agent-fleet-redis:6379]) and #1082 (prometheus client wrapper of the same failure).

Why this is noise, not an outage

957 has 808 events since 2026-07-01 and was burying the fleetq project. It is bursty, not chronic — the 100 most recent events fall into three windows:

Window Events
2026-07-21 19Z 76
2026-07-22 13Z 22
2026-08-03 12Z 2

Days of silence in between. Each burst lines up with a container recreate or redis config change (prod carries a .env.bak-redis-20260722075736 from the 07-22 window; the 08-03 pair is the deploy of #133). Culprit is /var/www/artisan — scheduler and queue workers reconnecting while redis is still coming up.

This is the same failure mode the filter already suppresses for Postgres one branch above:

// SQLSTATE[08006] — postgres connection_failure on container restart race.

Scope — what still reports

Deliberately narrow, because blanket-suppressing Redis errors would hide real defects:

  • Matches StreamInitException only. Predis throws it exclusively from StreamFactory during stream init, i.e. connect time. A read/write error on an already-established connection (the class fixed in 16c2b9f2) is untouched — there is a regression test asserting exactly that.
  • A sustained outage stays visible: HealthController pings Redis and flips /api/v1/health to 503 degraded. Verified in the controller, not assumed.
  • The prometheus branch matches by message because the client wraps the failure in its own storage exception class, which the instanceof check cannot catch.

Tests

3 added to BeforeSendFilterWebhookTest: drops StreamInitException, drops the prometheus wrapper, and keeps a ConnectionException read error. 9 passed. Pint + phpstan clean on the touched paths.

…art race

Sentry #957 accumulated 808 events since 2026-07-01 and buried the rest of
the fleetq project. It is not a chronic outage: the 100 most recent events
fall into three bursts (2026-07-21, -07-22, -08-03) with days of silence
between, each lining up with a container recreate / redis config change.
Culprit is /var/www/artisan — scheduler and queue workers reconnecting while
redis is still coming up.

Same shape as the existing SQLSTATE[08006] postgres filter, for the other
datastore. Scoped to Predis StreamInitException, which is thrown only from
StreamFactory during stream init, so a read/write error on an established
connection still reports (regression-tested).

Also drops the prometheus client's own wrapper of the same failure reaching
the /metrics scrape (#1082), which the class check cannot catch.

A sustained outage stays visible: HealthController pings Redis and flips
/api/v1/health to 503 degraded.
@escapeboy
escapeboy merged commit 0446478 into develop Aug 3, 2026
3 checks passed
@escapeboy
escapeboy deleted the fix/sentry-filter-redis-connect-race branch August 3, 2026 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant