feat: alert on node conntrack table saturation - #6190
Conversation
ConntrackFull (nf_conntrack table exhaustion) on DEV mgmt system nodes (AROSLSRE-1426) was only detectable after packets were already dropping, via kubelet/DNS/IMDS timeouts and kernel "table full" messages. Add a leading-indicator PrometheusRule that fires before saturation using node-exporter's node_nf_conntrack_entries / node_nf_conntrack_entries_limit (already ingested into the services Azure Monitor Workspace). The alert is ratio-based, so it is pool-agnostic and auto-adapts to each node's nf_conntrack_max (= 32768 x vCPU): NodeConntrackTableSaturation fires at >80% utilization sustained for 10m (severity warning / IcM Sev 3). Includes promtool test scenarios and the regenerated Azure Monitor rule group bicep. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Skipping CI for Draft Pull Request. |
There was a problem hiding this comment.
Pull request overview
Adds a new leading-indicator node-exporter based alert to detect Linux conntrack table saturation early on management cluster nodes, along with promtool test coverage and regenerated Azure Monitor rule group output.
Changes:
- Add
NodeConntrackTableSaturationalert tokube-node-rulesusingnode_nf_conntrack_entries / node_nf_conntrack_entries_limit. - Add promtool scenarios to validate firing/quiet behavior, including a <10m spike case.
- Regenerate
generatedPrometheusAlertingRules.bicepto include the new alert in the Azure Monitor Prometheus rule group.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| observability/alerts/kubeNode-prometheusRule.yaml | Adds the new conntrack saturation alert rule. |
| observability/alerts/kubeNode-prometheusRule_test.yaml | Adds promtool tests for the new alert behavior. |
| dev-infrastructure/modules/metrics/rules/generatedPrometheusAlertingRules.bicep | Regenerates Azure Monitor Prometheus rule group output to include the new alert. |
|
/retest |
2 similar comments
|
/retest |
|
/retest |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: raelga, roivaz The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Why
On DEV e2e-parallel management clusters, the single
aks-systemnode hits Linux conntrack table exhaustion (nf_conntrack: table full, dropping packet—ConntrackFull), timing out kubelet/DNS/IMDS on that node and degrading node health (AROSLSRE-1426).The follow-up ask on that issue: can we detect this before it bites in the future? Today it only surfaces after packets are already dropping (Kubernetes
ConntrackFullevents / kernel messages — a lagging symptom).What
Add a leading-indicator alert,
NodeConntrackTableSaturation, to thekube-node-rulesgroup. It uses node-exporter'snode_nf_conntrack_entries / node_nf_conntrack_entries_limit, which is already ingested into the services Azure Monitor Workspace (nodeexporter = true, empty keep-list,minimalingestionprofile = false) — no scrape/config change needed.Being ratio-based, it is pool-agnostic and auto-adapts to each node's
nf_conntrack_max(=32768 × vCPU), so it flags a saturating node (e.g. the small D4 system node) well before exhaustion, without noise from the largeruserswftpool.Changes
observability/alerts/kubeNode-prometheusRule.yaml— new alertobservability/alerts/kubeNode-prometheusRule_test.yaml— 3 promtool scenarios (fires >80%/10m; quiet when low; quiet on a <10m spike)dev-infrastructure/modules/metrics/rules/generatedPrometheusAlertingRules.bicep— regenerated Azure Monitor rule groupValidation
make run-sl-services(intooling/prometheus-rules) runs the promtool tests green and regenerates the bicep;make fmt-devinfraapplied.