feat(observability): deploy PrometheusMetricsAbsentPerCluster dead-man's-switch (AROSLSRE-1551) - #6186
Conversation
…n's-switch (AROSLSRE-1551) The PrometheusMetricsAbsentPerCluster alert already exists and is unit-tested in observability/alerts/prometheus-prometheusRule.yaml (group prometheus-wip-rules), but it was never listed in includedAlertsByGroup in alerts-msft-services.yaml, so the generator excluded it from generatedMsftPrometheusAlertingRules.bicep and it was never deployed to Azure Monitor / IcM. As a result, when the prom-agent on int-uksouth-mgmt-1 OOM-crashlooped, the cluster went dark for ~37h with no page. This adds the alert to the MSFT allowlist so the per-cluster total-absence detector (up absent 10m while present within 7d) is generated into the deployed bicep and pages at Sev2.
There was a problem hiding this comment.
Pull request overview
This PR makes the existing, unit-tested PrometheusMetricsAbsentPerCluster “dead-man’s-switch” alert actually deploy to the Microsoft Azure Monitor/IcM alerting stack by allowlisting it for MSFT rule generation. This closes the gap where a cluster could go completely silent on metrics without paging.
Changes:
- Add
PrometheusMetricsAbsentPerClusterfrom theprometheus-wip-rulesgroup toincludedAlertsByGroupinobservability/alerts-msft-services.yaml. - Regenerate the MSFT Prometheus alerting Bicep so the alert is emitted as a new Azure Monitor Prometheus rule group (
msft-prometheus-wip-rules) withfor: PT10Mand Sev2 mapping.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
observability/alerts-msft-services.yaml |
Allowlists PrometheusMetricsAbsentPerCluster so it gets emitted into the MSFT-generated Azure Monitor/IcM rules. |
dev-infrastructure/modules/metrics/rules/generatedMsftPrometheusAlertingRules.bicep |
Adds the generated msft-prometheus-wip-rules rule group containing PrometheusMetricsAbsentPerCluster with IcM actions and runbook link. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: janboll, raelga The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test e2e-parallel |
Why
When the
prom-agentPrometheusAgent onint-uksouth-mgmt-1OOM-crashlooped, the cluster stopped shipping all metrics for ~37h and no alert paged. Investigation showed the dead-man's-switch we would want —PrometheusMetricsAbsentPerCluster— already exists and is unit-tested inobservability/alerts/prometheus-prometheusRule.yaml(groupprometheus-wip-rules):It fires (severity
critical,for: 10m) precisely when a cluster reported within the last 7 days but has gone silent for 10 minutes — exactly the INT scenario. But the group was never added toincludedAlertsByGroupinalerts-msft-services.yaml, which is the explicit allowlist controlling what the generator emits intogeneratedMsftPrometheusAlertingRules.bicep. So it was excluded from the deployed Azure Monitor / IcM rules and never paged.What
PrometheusMetricsAbsentPerCluster(groupprometheus-wip-rules) toincludedAlertsByGroupinobservability/alerts-msft-services.yaml.dev-infrastructure/modules/metrics/rules/generatedMsftPrometheusAlertingRules.bicep(make -C observability alerts). The alert now emits as Azure groupmsft-prometheus-wip-ruleswith IcM actions,for: PT10M, Sev2, runbookdocs/alerts/Prometheus.md.Scope is intentionally limited to the single total-absence dead-man's-switch. The remaining
prometheus-wip-rulesalerts (uptime SLO, pending/failed rate, etc.) stay WIP and can be graduated separately.Jira: AROSLSRE-1551 (sub-task of AROSLSRE-1549, INT Prometheus resilience).