ops(dev): pin backend to dev-pool, off spot (closes #678)#679
Merged
Conversation
The backend serves the user-facing API — the one workload a spot reclamation must never interrupt (two user-visible 503s on 2026-07-12 traced to backend co-located on spot). It requests only 200m/512Mi and fits existing dev-pool headroom (~1150m free), so pinning it to the on-demand pool closes the abrupt-reclamation gap for ~$0 — cheaper than a 2nd replica, which would stay on spot and need anti-affinity to help. ADR-015 amendment: spot is for restartable/batch (agents, LiteLLM); the user-facing API + Redis belong on dev-pool. Redis moved in #676; this finishes it. Zero-downtime deployable via #676's RollingUpdate. Closes #678 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MnRCAFgjrrGZxo9VRCmCm9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last piece of "routine cluster events never cause user-visible downtime," and the cheap alternative to replicas=2 (which Sam correctly flagged as expensive).
What: backend Deployment gets
pool: devnodeSelector +pool=dev:NoScheduletoleration in values-dev — same shape Redis got in #676.replicaCountstays 1.Why ~$0: backend requests only 200m CPU / 512Mi and the dev-pool node has ~1150m free, so it fits on hardware already paid for — no new node. Removes the user-facing API from preemptible hardware entirely, so an abrupt spot reclamation (the trigger for two 503s on 2026-07-12) can't touch it. Cheaper and simpler than a 2nd replica, which would stay on spot and need anti-affinity + a PDB to get the same resilience.
Principle (ADR-015 amendment): spot is right for restartable/batch workloads (agents, LiteLLM). The user-facing API and Redis belong on the stable pool. Redis moved in #676; this finishes the job.
Zero-downtime deployable thanks to #676's RollingUpdate. Verification after merge+deploy: confirm the backend pod lands on a dev-pool node and health stays 200 through the reschedule.
Closes #678
🤖 Generated with Claude Code
https://claude.ai/code/session_01MnRCAFgjrrGZxo9VRCmCm9