feat: ARO-25981 - Adding Prometheus Alerts For etcd Metrics

[kevinobriendotca]

What

Adding Prometheus alerts for etcd metrics exposed via the SRE metrics set as described in ARO-25981.

Alerts follow the design defined in the ARO HCP Alerting Recommendation ADR.

See https://github.com/openshift-online/architecture/pull/79 for details.

18 alerts total in three separate burn-rate tiers.

│ Alert │ Fast (1h/5m) │ Medium (6h/30m) │ Slow (3d) │

│ EtcdBackendCommitDurationHigh │ for: 5m, rate[5m] │ for: 30m, rate[30m] │ for: 6h, rate[6h] │

│ EtcdDatabaseInUseSizeHigh │ for: 5m │ for: 30m │ for: 6h │

│ EtcdDatabaseSizeExceeded │ for: 5m │ for: 30m │ for: 6h │

│ EtcdFrequentLeaderChanges │ for: 5m │ for: 30m │ for: 6h │

│ EtcdPeerRoundTripTimeHigh │ for: 5m, rate[5m] │ for: 30m, rate[30m] │ for: 6h, rate[6h] │

│ EtcdWALFsyncDurationHigh │ for: 5m, rate[5m] │ for: 30m, rate[30m] │ for: 6h, rate[6h] │

Prometheus rules and test file generated with the help of claude-sonnet-4-6.

Why

We are currently lacking alerts for etcd performance degradation scenarios. Alerts are only being added for the etcd instances in the customer HCP clusters.

Testing

Automated tests that run via make alerts are passing and generated alerts in the bicep files are deployed successfully via make personal-dev-env.

Confirmed alerts are firing via Azure Portal and a test email action group. Example of firing alert email below.

Alert name etcd-availability/EtcdPeerRoundTripTimeHigh1h5m
Severity Sev3
Monitor condition Fired
Affected resource services-usw3keob
Resource type microsoft.monitor/accounts
Resource group hcp-underlay-pers-xyz
Description etcd instance etcd-6a29af2d5dac860001bca2bb-dgrv4746q2 has 99th percentile peer round-trip time of 0.10122971428571428s (threshold: 100ms). Network latency between peers may be affecting cluster performance.
Monitoring service Prometheus
Signal type Metric
Fired time June 17, 2026 3:28 UTC
Alert ID f45d3e02-32cd-bd71-1a3f-da9f015e000a - .../providers/Microsoft.AlertsManagement/prometheusRuleGroups/etcd-availability

PR Checklist

• PR is scoped to a single task (no mixed concerns)
• Title follows Conventional Commits format
• Summary explains the "Why" behind the change
• Linked to relevant ticket/issue
• Screenshots included (if graph/UI/metrics changes)
• Self-reviewed the diff
• CI/CD checks are passing (ignore Tide)
• Draft PR used for WIP (if applicable)
• Commit history is clean (rebased/squashed)
• Tricky code blocks are commented
• Specific reviewers tagged
• All comment threads resolved before merge

[openshift-ci]

Hi @kevinobriendotca. Thanks for your PR.

I'm waiting for a Azure member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR tightens some Kubernetes control-plane alert firing durations and introduces a new set of etcd availability/performance PrometheusRule alerts along with promtool tests, then wires the new rule file into existing alert bundles.

Changes:

• Reduce for durations (and corresponding annotations) for a few Kubernetes alerts (e.g., 30m → 15m, 90m → 15m).
• Add a new etcd-availability PrometheusRule group with tiered alerting and a new promtool test suite.
• Include the new etcd rule file in SRE/HCP and SL services rule bundles.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
observability/alerts/kubernetesControlPlane-prometheusRule.yaml	Shortens alert for durations / descriptions for select Kubernetes alerts.
observability/alerts/etcd-availability-prometheusRule.yaml	Adds new tiered etcd performance/availability alerts.
observability/alerts/etcd-availability-prometheusRule_test.yaml	Adds promtool unit tests for the new etcd alert rules.
observability/alerts-sre-hcps.yaml	Registers the new etcd PrometheusRule in the SRE/HCP rules list.
observability/alerts-sl-services.yaml	Registers the new etcd PrometheusRule in the SL services rules list and reorders entries.

[kevinobriendotca]

Hmm, not sure what you mean here, Copilot. There are no "|" characters in the file and running "promtool test rules --debug etcd-availability-prometheusRule_test.yaml" shows the expected output for the passing tests. The tests are also passing via make alerts.

[kevinobriendotca]

This is an upstream change pulled in via make alerts from https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/e02554298cb62b5533f3407c8eacc664e80bc74b/manifests/kubernetesControlPlane-prometheusRule.yaml.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

observability/alerts-sl-services.yaml:12

• rulesFolders includes duplicate entries for backend, fleet, and frontend (these files are each listed twice). Even if generation currently dedupes, this is confusing and risks accidental drift if one instance is edited later.

Remove the duplicate entries and keep a single cluster-service-slo-rules.yaml entry here, then re-run make alerts to regenerate the generated bicep outputs.

  - ../admin/alerts/admin-prometheusRule.yaml
  - ../backend/alerts/backend-prometheusRule.yaml
  - ../cluster-service/alerts/cluster-service-slo-rules.yaml
  - ../fleet/alerts/fleet-prometheusRule.yaml
  - ../frontend/alerts/frontend-prometheusRule.yaml

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

observability/alerts-rp-hcps.yaml:16

• PR description says etcd alerts are being added for backend management + service clusters as well as HCP clusters, but this PR wires the new etcd rules only into the RP HCP alert config (the only reference to alerts/etcd-availability-prometheusRule.yaml is here). Either add the etcd alert group into the other intended alert configs/lanes, or update the PR description to reflect the actual scope.

  rulesFolders:
  - alerts/etcd-availability-prometheusRule.yaml

[swiencki]

Three items from reviewing against the ADR:

1. runbook_url: Should use https://aka.ms/arohcp-runbook-etcd short-link per ADR, not the full eng.ms path.

2. Fragmentation/db-size/leader-changes: These are gauge/level metrics, not event rates, so they should be simple threshold alerts, not MWMBR (the same expr fires identically at all 3 tiers).

3. component: fleet: etcd runs per-HCP, should this be component: hcp?

[Copilot]

🟡 Not ready to approve

Several etcd histogram-based alerts aggregate away resource_id while still templating it into annotations/correlation IDs, producing empty identifiers and necessitating corresponding rule/test updates.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (6)

observability/alerts/etcd-availability-prometheusRule.yaml:203

• Same issue here: sum by (cluster, instance, le) drops resource_id, but annotations/correlation IDs expect it. Add resource_id to the aggregation labels so the alert series retains it.

        histogram_quantile(0.99, sum by (cluster, instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{subscription_id!~"974ebd46-8ad3-41e3-afef-7ef25fd5c371|e8c5a115-842d-4d7e-98ad-cfb2c50b209e|e627aa70-36a3-40b0-8e68-975269e39d7b|6ed122d1-7e03-4a01-baae-9020abf350d4|64f0619f-ebc2-4156-9d91-c4c781de7e54|dee2f1be-a999-4e19-b027-221e7adaf7d3|8d696692-794f-4cdb-ba25-9250c9e9ec4c|ec435068-e722-475f-8504-c91b72a5dc51|403d9de9-132b-4974-94a5-5b78bdfa191e"}[5m]))) > 0.1
        and
        histogram_quantile(0.99, sum by (cluster, instance, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket{subscription_id!~"974ebd46-8ad3-41e3-afef-7ef25fd5c371|e8c5a115-842d-4d7e-98ad-cfb2c50b209e|e627aa70-36a3-40b0-8e68-975269e39d7b|6ed122d1-7e03-4a01-baae-9020abf350d4|64f0619f-ebc2-4156-9d91-c4c781de7e54|dee2f1be-a999-4e19-b027-221e7adaf7d3|8d696692-794f-4cdb-ba25-9250c9e9ec4c|ec435068-e722-475f-8504-c91b72a5dc51|403d9de9-132b-4974-94a5-5b78bdfa191e"}[1h]))) > 0.1

observability/alerts/etcd-availability-prometheusRule.yaml:250

• Same issue here: sum by (cluster, instance, le) drops resource_id, but annotations/correlation IDs expect it. Add resource_id to the aggregation labels so the alert series retains it.

        histogram_quantile(0.99, sum by (cluster, instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{subscription_id!~"974ebd46-8ad3-41e3-afef-7ef25fd5c371|e8c5a115-842d-4d7e-98ad-cfb2c50b209e|e627aa70-36a3-40b0-8e68-975269e39d7b|6ed122d1-7e03-4a01-baae-9020abf350d4|64f0619f-ebc2-4156-9d91-c4c781de7e54|dee2f1be-a999-4e19-b027-221e7adaf7d3|8d696692-794f-4cdb-ba25-9250c9e9ec4c|ec435068-e722-475f-8504-c91b72a5dc51|403d9de9-132b-4974-94a5-5b78bdfa191e"}[5m]))) > 0.01
        and
        histogram_quantile(0.99, sum by (cluster, instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket{subscription_id!~"974ebd46-8ad3-41e3-afef-7ef25fd5c371|e8c5a115-842d-4d7e-98ad-cfb2c50b209e|e627aa70-36a3-40b0-8e68-975269e39d7b|6ed122d1-7e03-4a01-baae-9020abf350d4|64f0619f-ebc2-4156-9d91-c4c781de7e54|dee2f1be-a999-4e19-b027-221e7adaf7d3|8d696692-794f-4cdb-ba25-9250c9e9ec4c|ec435068-e722-475f-8504-c91b72a5dc51|403d9de9-132b-4974-94a5-5b78bdfa191e"}[1h]))) > 0.01

observability/alerts/etcd-availability-prometheusRule_test.yaml:20

• The promtool tests currently assume cluster and resource_id are absent (note the leading spaces / empty segments in expected annotations). If you keep resource_id in the alert text/correlation IDs, the test input series should include representative cluster and resource_id labels so expectations validate the real alert payloads.

# Scenario 1: High commit duration — all three tiers fire
- interval: 1m
  input_series:
  - series: 'etcd_disk_backend_commit_duration_seconds_bucket{instance="etcd-0",le="0.001"}'
    values: "0+10x370"

dev-infrastructure/modules/metrics/rp-rules.bicep:27

• rp-rules.bicep now includes both RP service alert rules and HCP alert rules, but it only takes a single azureMonitoringWorkspaceId parameter (documented as services/AKS metrics). This would deploy HCP rules to the wrong workspace if this module is used. Since monitoring.bicep now deploys HCP rules via rp-hcp-rules.bicep, consider removing the HCP rules from this combined module (or reworking it to accept a separate HCP workspace ID).

module rpHcpGeneratedAlerts 'rules/generatedRPHCPPrometheusAlertingRules.bicep' = {
  name: 'generatedRPHCPPrometheusAlertingRules'
  params: {
    azureMonitoring: azureMonitoringWorkspaceId
    actionGroups: actionGroups
    severityCeiling: severityCeiling
  }
}

observability/alerts/etcd-availability-prometheusRule.yaml:27

• These histogram-based alerts reference $labels.resource_id in summary/description, but the sum by (cluster, instance, le) aggregation drops the resource_id label, so resource_id (and correlation IDs derived from it) will be empty at runtime. Include resource_id in the aggregation labels so it propagates through histogram_quantile.

This issue also appears in the following locations of the same file:

• line 201
• line 248

        histogram_quantile(0.99, sum by (cluster, instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{subscription_id!~"974ebd46-8ad3-41e3-afef-7ef25fd5c371|e8c5a115-842d-4d7e-98ad-cfb2c50b209e|e627aa70-36a3-40b0-8e68-975269e39d7b|6ed122d1-7e03-4a01-baae-9020abf350d4|64f0619f-ebc2-4156-9d91-c4c781de7e54|dee2f1be-a999-4e19-b027-221e7adaf7d3|8d696692-794f-4cdb-ba25-9250c9e9ec4c|ec435068-e722-475f-8504-c91b72a5dc51|403d9de9-132b-4974-94a5-5b78bdfa191e"}[5m]))) > 0.025
        and
        histogram_quantile(0.99, sum by (cluster, instance, le) (rate(etcd_disk_backend_commit_duration_seconds_bucket{subscription_id!~"974ebd46-8ad3-41e3-afef-7ef25fd5c371|e8c5a115-842d-4d7e-98ad-cfb2c50b209e|e627aa70-36a3-40b0-8e68-975269e39d7b|6ed122d1-7e03-4a01-baae-9020abf350d4|64f0619f-ebc2-4156-9d91-c4c781de7e54|dee2f1be-a999-4e19-b027-221e7adaf7d3|8d696692-794f-4cdb-ba25-9250c9e9ec4c|ec435068-e722-475f-8504-c91b72a5dc51|403d9de9-132b-4974-94a5-5b78bdfa191e"}[1h]))) > 0.025

cluster-service/alerts/cluster-service-slo-rules.yaml:394

• The new component: backend label value on Cluster Service SLO alerts is inconsistent with how other service rule groups label component (e.g., backend/alerts/* use component: backend, frontend uses component: frontend, maestro uses component: maestro). If component is used for routing or dashboards, these alerts will be misclassified; consider using a Cluster Service-specific value (and updating the test expectations).

      for: 5m
      labels:
        component: backend
        long: 6h
        severity: warning

• Files reviewed: 18/19 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

dev-infrastructure/modules/metrics/rp-rules.bicep currently wires RP-HCP alert rules to the services workspace parameter, which is likely incorrect and should be corrected/removed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

• Files reviewed: 18/19 changed files
• Comments generated: 1
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[kevinobriendotca]

Three items from reviewing against the ADR:

1. **runbook_url**: Should use `https://aka.ms/arohcp-runbook-etcd` short-link per ADR, not the full eng.ms path.

2. **Fragmentation/db-size/leader-changes**: These are gauge/level metrics, not event rates, so they should be simple threshold alerts, not MWMBR (the same expr fires identically at all 3 tiers).

3. **`component: fleet`**: etcd runs per-HCP, should this be `component: hcp`?

1. Updated in the code. Redirect to be added.
2. Done.
3. I didn't see hcp as an option in the list of component values in Steve's comment when he added the mandatory component field validation. It doesn't look like it compares the component value to a list, so switched it to hcp until I hear different.

[Copilot]

🟡 Not ready to approve

There are unresolved alert-routing/scoping issues (misleading component labeling for Cluster Service SLO alerts and potentially unsafe scoping in rp-rules.bicep) that could impact operational behavior if merged as-is.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (9)

dev-infrastructure/modules/metrics/rp-rules.bicep:27

• rp-rules.bicep now wires the HCP alert rules module to azureMonitoringWorkspaceId, whose description says it is for services/AKS level metrics. This makes it easy to accidentally deploy HCP alerts to the wrong Azure Monitor Workspace if this module is reused in the future (the HCP lane uses hcpAzureMonitoringWorkspaceId in templates/monitoring.bicep). Consider keeping rp-rules.bicep service-scoped now that RP service/HCP rules are deployed via dedicated modules.

module rpHcpGeneratedAlerts 'rules/generatedRPHCPPrometheusAlertingRules.bicep' = {
  name: 'generatedRPHCPPrometheusAlertingRules'
  params: {
    azureMonitoring: azureMonitoringWorkspaceId
    actionGroups: actionGroups
    severityCeiling: severityCeiling
  }
}

dev-infrastructure/modules/metrics/rp-service-rules.bicep:1

• This file comment says these alerts are "reserved for future" use, but dev-infrastructure/templates/monitoring.bicep now deploys rp-service-rules.bicep in the RP lane. Please update the comment so it accurately reflects current usage.

// Alerts and Recording Rules reserved for future ARO-RP-owned user journey alerts.

dev-infrastructure/modules/metrics/rp-hcp-rules.bicep:1

• This file comment says these alerts are "reserved for future" use, but dev-infrastructure/templates/monitoring.bicep now deploys rp-hcp-rules.bicep to publish HCP etcd alerts. Please update the comment so it accurately reflects current usage.

// Alerts and Recording Rules reserved for future ARO-RP-owned user journey alerts.

cluster-service/alerts/cluster-service-slo-rules.yaml:406

• These SLO alerts are for the Cluster Service (namespace="clusters-service", summaries reference "Cluster Service"), but the newly added component: backend label suggests a different owning component and may misroute/misclassify alerts. Please confirm the intended component label and align it consistently with other SLO alerting rules.

        component: backend

cluster-service/alerts/cluster-service-slo-rules.yaml:437

• These SLO alerts are for the Cluster Service (namespace="clusters-service"), but the newly added component: backend label may misroute/misclassify alerts. Please confirm the intended component label and align it consistently.

        component: backend

cluster-service/alerts/cluster-service-slo-rules.yaml:450

• These SLO alerts are for the Cluster Service (namespace="clusters-service"), but the newly added component: backend label may misroute/misclassify alerts. Please confirm the intended component label and align it consistently.

        component: backend

cluster-service/alerts/cluster-service-slo-rules.yaml:481

• These SLO alerts are for the Cluster Service (namespace="clusters-service"), but the newly added component: backend label may misroute/misclassify alerts. Please confirm the intended component label and align it consistently.

        component: backend

cluster-service/alerts/cluster-service-slo-rules.yaml:494

• These SLO alerts are for the Cluster Service (namespace="clusters-service"), but the newly added component: backend label may misroute/misclassify alerts. Please confirm the intended component label and align it consistently.

        component: backend

cluster-service/alerts/cluster-service-slo-rules.yaml:392

• These SLO alerts are for the Cluster Service (namespace="clusters-service", summaries reference "Cluster Service"), but the newly added component: backend label suggests a different owning component and may misroute/misclassify alerts. Other SLO alert groups in observability/alerts/*-slo-*.yaml use component: slo at the group level (e.g. observability/alerts/access-cluster-slo-prometheusRule.yaml:29-30). Please confirm the intended component label here and align it consistently.

This issue also appears in the following locations of the same file:

• line 406
• line 437
• line 450
• line 481
• line 494

        component: backend

• Files reviewed: 18/19 changed files
• Comments generated: 0 new
• Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.