fix(repository_hub): cache GitHub max-statuses 422 to stop rate-limit drain

[cchristous]

Summary

Fixes a long-standing GitHub-API rate-limit drain caused by POST /repos/:owner/:repo/statuses/:sha retrying forever against SHA+context combinations that have already reached GitHub's documented 1000-statuses-per-SHA limit.

In one self-hosted deployment we observed this single bug burning ~1,400 requests / 2 days on a single stuck SHA — ≈26% of the staging GitHub App's hourly budget. GitHub audit-log analysis confirmed 1,397 of 1,400 status posts were all to the same :repository_id/statuses/:sha, all returning 422.

What was broken

Two issues compounded:

1. is_max_statuses_response? matcher mismatched the real response shape. GitHub returns the validation error with errors as a string (see @real_422_body fixture):

   {
     "errors": "Validation failed: This SHA and context has reached the maximum number of statuses.",
     "message": "Validation Failed",
     "status": "422"
   }

   fetch_errors/1 expected errors to be a list of objects, so it returned [] and is_max_statuses_response? always answered false. Every max-statuses 422 was therefore treated as a fatal :precondition error and propagated upstream — where the caller (in this case github_notifier) retries with a fresh ppl_id, so the dedup cache key changes and the next pipeline run does the same 422 dance.

2. No caller-side prevention. Even with the matcher working, every new pipeline run still hits GitHub. The 422 response costs a rate-limit unit just like a 201, so the matcher recognising it after the fact didn't save any budget.

The matcher has been broken since the very first OSS commit — git log -L:'is_max_statuses_response?':repository_hub/lib/repository_hub/clients/github_client.ex confirms it has never been modified.

What this PR does

repository_hub/:

• Rewrites is_max_statuses_response? to gate on message == "Validation Failed" and substring-match "maximum number of statuses" against either a string or a list of error objects (errors_as_text/1).
• Adds RepositoryHub.MaxStatusesCache — a per-pod ETS-backed GenServer keyed by {owner, repo, sha, context} with a 24h TTL. Before every create_build_status call we consult the cache; on a hit we short-circuit and return wrap(%{}) — no POST, no rate-limit pre-flight, no rate-limit unit consumed. After detecting a true max-statuses 422 we populate the cache.
• Moves the 422 branch into its own handle_422_create_build_status/2 clause so the max-statuses matcher cannot inadvertently fire on 307/404/403 responses.
• Adds a canary log_warn when a 422 carries message: "Validation Failed" but the canonical substring isn't found — this will surface in logs the moment GitHub rewords the error, so we can update the matcher before the rate-limit drain bug silently regresses.
• Adds observability:
  • github_api.create_build_status.max_statuses_cache_hit Watchman counter
  • github_api.max_statuses_cache.table_missing Watchman counter + Logger.warning (the ETS table being absent is treated as a soft failure that callers should still survive, but it's now alertable instead of silent)
  • Includes repo_owner/repo_name@commit_sha ctx=… in the existing log_error lines.

Why this is safe

• Behaviour on success (HTTP 201) is unchanged.
• Behaviour on non-max-statuses 422 / 403 / 404 / 307 is unchanged — those still log and fail_with(:precondition, …). Only the specific "maximum number of statuses" 422 is treated as success and cached. New tests cover 403/404 with a misleading body to lock this down.
• wrap(%{}) short-circuit shape matches the existing max-statuses success path — the sole caller (adapters/github/create_build_status_action.ex) only pattern-matches {:ok, _} and discards the payload.
• The cache is in-process ETS — no new external dependency, no new failure modes. Each pod owns its own cache; worst case after a pod restart is one wasted request per maxed SHA+context before the cache is repopulated.
• 24h TTL — stale entries naturally fall out; nothing pins memory for inactive SHAs forever.
• :ets.whereis/1 is checked before every read/write — if the GenServer is down, callers degrade to "no cache" (the original behaviour, with observability) instead of crashing.

Tests

12 new tests across two files. All pre-existing github_client_test.exs (8 tests) and create_build_status_action_test.exs (4 tests) continue to pass.

New tests cover:

• The current string-form errors shape (the real bug repro).
• The legacy list-of-objects errors shape (defensive — module supports both).
• A 422 with message: "Validation Failed" but an unrelated errors body is NOT cached and still surfaces as :precondition.
• 403 and 404 in the same case-branch as 422 are NEVER put through the matcher and never populate the cache, even with a body that contains the canonical max-statuses substring.
• After the first 422, the second call short-circuits before both Tentacat.Repositories.Statuses.create/5 and the Tentacat.get("rate_limit", _) pre-flight — the whole point of the fix is to spare every rate-limit unit.
• Cache hit/miss for all four key components (owner, repo, sha, context).
• Expired TTL entries are treated as misses.

Test plan

• mix test test/repository_hub/clients/{max_statuses_cache,github_client_max_statuses,github_client}_test.exs test/repository_hub/adapters/github/create_build_status_action_test.exs — 30 tests, 0 failures.
• mix format --check-formatted
• mix compile --warnings-as-errors
• (vendor CI) full repository_hub suite once this PR opens. Note: the Policy::MixAudit check flags pre-existing CVEs in cowboy, hackney, and decimal in mix.lock — unrelated to this change.