feat(observability): add SWIFT networking user journey monitoring (ARO-25382) - #6359
feat(observability): add SWIFT networking user journey monitoring (ARO-25382)#6359ventifus wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ventifus The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hi @ventifus. Thanks for your PR. I'm waiting for a Azure member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
…O-25382)
Add CNS PodMonitor, recording rules, multi-window multi-burn-rate alert
rules, and working draft documentation for the SWIFT networking user
journey (ARO-25382).
- Add azure-cns.podmonitor.yaml to scrape CNS metrics from management
cluster SWIFT pods (ARO-25919)
- Add router:startup_latency:* recording rules measuring router pod
startup latency as the journey-level SWIFT SLI, using kube-state-metrics
- Add userJourneySwiftLatencyP99{1h5m,6h30m,3d} alerts routing to the
new RP per-HCP lane; slow-burn (3d) uses 6h window, name reflects burn
rate (budget exhausts in ~3 days at this rate) (ARO-25978)
- Add alerts-rp-hcps.yaml — first RP per-HCP alert lane in the repo,
wired into rp-rules.bicep and tooling/prometheus-rules/Makefile
- Add working draft docs in docs/drafts/ for co-review before publishing
to eng.ms/ADO (runbook, TSG, SLI/SLO, dashboard, alerting design)
Alert naming follows ADR-001: userJourneySwift* (full camelCase).
Severity labels use numeric strings. runbook_url links to the runbook
per the alert-runbook-TSG chain.
6049995 to
98bedfc
Compare
1 participant
https://redhat.atlassian.net/browse/ARO-25382
Summary
azure-cns.podmonitor.yamlto scrape CNS metrics from management cluster SWIFT pods (ARO-25919)router:startup_latency:*recording rules measuring router pod ContainerCreating duration as the journey-level SWIFT SLI, using kube-state-metricsuserJourneySwiftLatencyP99{1h5m,6h30m,3d}multi-window multi-burn-rate alerts per ADR-001 (ARO-25978)alerts-rp-hcps.yaml— first RP per-HCP alert lane in the repo, wired intorp-rules.biceprun-rp-hcpsMakefile target; runmake alertsandmake recording-rulesto regenerate BicepWhy
SWIFT networking provides the private network path between ARO HCP customer VNets and hosted control planes. When it fails, worker nodes go NotReady, webhooks break, and cert rotations fail against private Key Vaults. This change adds the observability layer required by the SWIFT user journey runbook:
alerts-rp-hcps.yamlis the first RP per-HCP lane. The comment inrp-rules.bicep("reserved for future ARO-RP-owned user journey alerts") anticipated exactly this addition.userJourneySwift*(full camelCase) per ADR-001 Section 3.Test plan
make alertsandmake recording-rulespass; all promtool tests greencx_pending_programming_ips_v2) confirmed flowing to theservices-usw3adenAMW after deploying the PodMonitor in persocm-arohcppers-*namespaces confirmed routing to the HCP AMW (630kube_pod_status_phaseresults from a provisioned HCP control plane)arohcp_swift_networking_slo_recording_rulesandarohcp_swift_networking_alertsrule groups confirmed deployed in persrouter:startup_latency:p99live data validation — requires a fully SWIFT-provisioned HCP (PodNetwork/PNI configured). The pers cluster creation path does not set up SWIFT networking. Recording rule infrastructure and KSM routing are confirmed correct; end-to-end validation with live router pod data is a follow-up.