fix(mcp): stop rejecting valid API keys on non-200 upstream statuses

[Zernio-Elean]

Summary

• The hosted MCP server reported valid Zernio API keys as invalid_token, telling users to clear their credentials and re-register, whenever zernio.com answered with anything other than a 200.
• 402 and 429 prove the key is valid. The Zernio API emits them only after authenticate() has already resolved the credential, so treating them as auth failures inverted a distinction the API was built to expose.
• Auth failed closed with no caching, unlike the app-side path, so a transient upstream blip locked users out for as long as it lasted.

The bug

src/late/mcp/auth.py did return response.status_code == 200, collapsing every non-200 into "invalid token", and except Exception: return False swallowed the 5s timeout. verify_token then returned None, and fastmcp turned that into the 401 invalid_token challenge.

Measured on the MCP's own verification traffic (GET /api/v1/accounts, python-httpx), Aug 1-3 2026:

Upstream status	False rejections	Reported to the user as
402 Payment Required	32,190	invalid_token
429 Too Many Requests	7,034	invalid_token
500	878	invalid_token
504 (one 30-min window)	2,638	invalid_token

One customer's valid key was rejected for roughly 26 hours (Aug 1 00:16 to Aug 2 15:46). The 429s were self-inflicted: verifying on every single request burned the caller's own rate-limit bucket.

Changes

• Verification enum (VALID / INVALID / UNKNOWN) replaces the bool that was the bug.
• _classify: 200, 402, 429 -> VALID; 401, 403 -> INVALID; everything else -> UNKNOWN.
• Positive-only verification cache: sha256 keys, 10,000-entry LRU cap, 60s fresh window (mirrors the 60s TTL the API itself keeps on apikey:{keyHash}), 1h grace window used only on UNKNOWN.
• UNKNOWN for a token not in the cache raises AuthenticationError, which surfaces as HTTP 400 with a truthful message, never the 401 that triggers credential-clearing.

Safety

• An unseen token is never granted during an outage. The cache is written only on VALID, so the grace window can only ever re-honour a credential the API itself confirmed.
• Negatives are never cached: an INVALID verdict evicts the entry.
• A revoked key honoured within the grace window can still do nothing. server.py:_get_client re-presents the caller's own bearer to the Zernio API on every tool call, where it is rejected.
• The grace path never refreshes the timestamp, so the window cannot self-extend past 1h from the last real confirmation.

Behaviour note

The auth middleware is app-level, not route-level, so it wraps /health and the .well-known discovery routes too. During an upstream outage a request that carries a bearer to those routes now returns 400 where it previously returned 200. Railway's health check sends no Authorization header and is unaffected.

Testing

• tests/test_mcp_auth_verification.py, 7 tests. httpx.MockTransport only, no mocking of our own functions. auth.py previously had zero coverage.
• Covers the regression itself (402 must not return None), the security ruling (unknown token during an outage must raise, not grant), the grace window, INVALID evicting a cached entry, a genuinely bad key still being rejected, and the LRU bound.
• uv run pytest tests/ -q -> 202 passed, 14 skipped.
• uv run ruff check src tests -> clean.

Follow-ups (not in this PR)

• Alert on the Zernio token verification inconclusive: HTTP %s warning. It is the only signal if /api/v1/accounts is ever renamed or starts redirecting, which would make every token UNKNOWN. /health never exercises auth, which is why a 26h auth outage ran with a green container and no alert.
• Return 503 with Retry-After instead of 400 on the UNKNOWN path. Achievable by overriding get_middleware(), but it re-implements a framework method body and needs its own canary test.
• Delete scripts/smoketest_streamable_http.py. It has been dead since the FastMCP migration (it imports late.mcp.routes, which does not exist).

Crisp

https://app.crisp.chat/website/20dea5d6-a684-4c80-b097-2258b0b41421/inbox/session_f3d1f686-cadd-4832-bae3-32fa54047715/