fix(observability): harden INT mgmt Prometheus against single-agent OOM blackout - #6187
Merged
openshift-merge-bot[bot] merged 3 commits intoJul 22, 2026
Conversation
…OM blackout A single prom-agent OOM-crashloop blacked out metrics on int-uksouth-mgmt-1 for ~37h. Three durable mitigations, scoped to the INT mgmt clusters, plus one chart-wide HA improvement: - shards 2 -> 4 (INT mgmt): halves per-shard series and WAL, so a remote-write stall replay after a restart stays bounded instead of ballooning past node memory. - memory limit 4Gi (INT mgmt): caps a runaway pod so it OOM-kills itself rather than starving the ~11.8Gi infra node and taking neighbours down with it. - memory request 1.5Gi (INT mgmt): keeps all 8 pods (4 shards x 2 replicas) schedulable on the 2 infra nodes (4 x 1.5Gi = 6Gi/node). - per-shard replica anti-affinity (chart, all envs): add OnShard to the hostname topologySpreadConstraint so the two replicas of a shard spread across nodes. Soft (ScheduleAnyway) so it never blocks scheduling. Fixes an observed bug where both replicas of shard-1 landed on the same node. AROSLSRE-1553, AROSLSRE-1554
openshift-ci
Bot
requested review from
Ashish (ashishmax31) and
Anthony Vollmer (avollmer-redhat)
July 21, 2026 13:13
Contributor
There was a problem hiding this comment.
Pull request overview
Improves monitoring resilience for Microsoft INT management clusters by reducing single-prom-agent blast radius (via increased sharding + explicit memory request/limit) and strengthening shard-level HA placement (via shard-aware node spreading) in the Prometheus Helm chart.
Changes:
- INT-only: increase Prometheus agent sharding (2→4) and set memory requests/limits for mgmt Prometheus agents in the MSFT clouds overlay config.
- Chart-wide: add
additionalLabelSelectors: OnShardto the hostname topology spread constraint so replicas of the same shard preferentially land on different nodes. - Regenerate/commit Helm template fixtures reflecting the new topology spread constraint output.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
config/config.msft.clouds-overlay.yaml |
INT mgmt Prometheus agent sharding + memory requests/limits override. |
observability/prometheus/deploy/templates/prometheus.yaml |
Shard-aware hostname topology spread constraint to improve per-shard replica HA. |
observability/prometheus/testdata/zz_fixture_TestHelmTemplate_helmtest_svc_resources.yaml |
Updated generated fixture output for Prometheus chart changes. |
observability/prometheus/testdata/zz_fixture_TestHelmTemplate_helmtest_svc_resources_unset.yaml |
Updated generated fixture output for Prometheus chart changes. |
observability/prometheus/testdata/zz_fixture_TestHelmTemplate_helmtest_mgmt_resources.yaml |
Updated generated fixture output for Prometheus chart changes. |
observability/prometheus/testdata/zz_fixture_TestHelmTemplate_helmtest_mgmt_resources_unset.yaml |
Updated generated fixture output for Prometheus chart changes. |
dev-infrastructure/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_arohcp_monitor.yaml |
Updated generated fixture output for Prometheus chart changes. |
dev-infrastructure/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_arohcp_monitor.yaml |
Updated generated fixture output for Prometheus chart changes. |
Follow-on to the shards/anti-affinity change: give the INT mgmt prom-agents real room instead of running them at the edge of a 2-node infra pool. - infraAgentPool.minCount 1 -> 2: raises the guaranteed infra floor from 2 to 4 nodes (poolCount 2 x minCount 2). The autoscaler ceiling (maxCount 3/pool) is unchanged. Cost: +2 x Standard_D4ds_v5 (~+$330/mo for INT). - prometheusSpec.resources.limits.memory 4Gi -> 5Gi: a bit more OOM headroom per pod now that there is node capacity for it. - prometheusSpec.resources.requests.memory 1.5Gi -> 2Gi: keeps all 8 pods (4 shards x 2 replicas) comfortably schedulable across the 4 nodes (2 x 2Gi + ~2Gi infra baseline = ~6Gi/node of ~11.8Gi allocatable). INT mgmt only; svc/stg/prod unchanged. AROSLSRE-1555, AROSLSRE-1553
Collaborator
|
/lgtm |
infraAgentPool lives under mgmt.aks (aksConfig), not directly under mgmt. As mgmt.infraAgentPool it was an unknown field (aksConfig has additionalProperties: false) and the minCount bump would not apply. Move it under mgmt.aks to match the schema.
Collaborator
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: raelga, weherdh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Durable resilience mitigations for the INT management-cluster Prometheus agents, following the ~37h metrics blackout on
int-uksouth-mgmt-1(AROSLSRE-1530). A singleprom-agentOOM-crashloop took down all monitoring for that cluster; nothing alerted because the per-cluster dead-man's-switch was never deployed (fixed separately in #6186).This PR addresses the cause (unbounded single-agent memory + coarse sharding + a scheduling gap + a too-tight 2-node infra pool), scoped to the INT mgmt clusters, plus one small chart-wide HA fix.
Changes
INT mgmt (
config.msft.clouds-overlay.yaml,clouds.public.environments.int):infraAgentPool.minCount: 1 -> 2— raises the guaranteed infra floor from 2 to 4 nodes (poolCount 2 × minCount 2). Autoscaler ceiling (maxCount 3/pool) unchanged. Cost: +2 × Standard_D4ds_v5 ≈ +$330/mo for INT.shards: 2 -> 4— halves per-shard series/WAL, so a remote-write stall replay after a restart stays bounded instead of ballooning past node memory.resources.limits.memory: 4Gi -> 5Gi— caps a runaway pod so it OOM-kills itself rather than starving the infra node.resources.requests.memory: 1.5Gi -> 2Gi— keeps all 8 pods (4 shards × 2 replicas) comfortably schedulable across the 4 nodes.Chart (all envs),
observability/prometheus/deploy/templates/prometheus.yaml:additionalLabelSelectors: OnShardto thekubernetes.io/hostnametopologySpreadConstraint so the two replicas of each shard spread across nodes. Soft (ScheduleAnyway) so it can never block scheduling. Fixes an observed bug on INT where both replicas of shard-1 landed on the same infra node, defeating replica HA.Feasibility (live INT data)
Standard_D4ds_v5, ~11.8Gi allocatable each); agent memory effectively unbounded; 3 of 4 prom pods pinned to one node.Scope / safety
make -C observability verify,make verify-schema, andmake -C config materializepass.Rollout
INT mgmt ships via
sdp-pipelinesEV2 (mgmt-pipeline). TheminCountbump adds nodes (non-disruptive); no SKU change (which would be node-pool recreation).Tracking: AROSLSRE-1552, AROSLSRE-1553, AROSLSRE-1554, AROSLSRE-1555 (Story AROSLSRE-1549, INFRA epic AROSLSRE-798).