docs: add Maestro agent CloudEvents desync scenario and scale-down strategy - #6189
Conversation
…rategy Adds operational learnings from AROSLSRE-1520 (50 clusters stuck deleting in brazilsouth) to the stuck deletion and stale resource bundle runbooks: - Scenario 7: Maestro Agent ignores soft-deleted ResourceBundles due to CloudEvents sync bug (ARO-28432), causing ManifestWork recreation - Strategy 3: Scale down Maestro Agent before cleaning ManifestWorks to eliminate race condition vs. restart - GOTCHA: deleted_at field is at JSON root level, not .metadata.deleted_at - Port-forward guidance for reliable Maestro API inspection - Cross-references between the two runbooks
|
Skipping CI for Draft Pull Request. |
|
/approve |
There was a problem hiding this comment.
Pull request overview
Updates the ops runbooks to document a newly observed Maestro failure mode where the Maestro Agent continues recreating ManifestWorks despite ResourceBundles being soft-deleted (CloudEvents sync desync), and adds a safer remediation strategy that scales the agent down during cleanup. It also corrects and emphasizes that deleted_at is a root-level ResourceBundle JSON field (not under .metadata), which affects investigation queries.
Changes:
- Adds Scenario 7 to the stuck deletion runbook, with distinguishing criteria vs. existing Maestro bundle scenarios.
- Introduces Strategy 3 to scale down Maestro Agent during ManifestWork/AppliedManifestWork cleanup to avoid recreation races; renumbers the prior last-resort strategy.
- Adds cross-references and “gotcha” guidance about correctly querying
deleted_at, plus a port-forward inspection quick reference.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/ops/cleanup-stuck-cluster-deletion.md | Adds Scenario 7 + new Strategy 3 (agent scale-down cleanup), fixes deleted_at jq usage, and updates cross-references/quick references. |
| docs/ops/fix-maestro-stale-resource-bundle.md | Adds a related-scenarios/gotchas section, including deleted_at field location warning and a link to Scenario 7. |
Suppressed comments (3)
docs/ops/cleanup-stuck-cluster-deletion.md:329
- This uses
grep "${CLUSTER_ID}"which treats the cluster ID as a regex and can match unintended resources. Using fixed-string matching makes the cleanup commands safer.
for amw in $(kubectl get appliedmanifestwork -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep "${CLUSTER_ID}"); do
docs/ops/cleanup-stuck-cluster-deletion.md:340
- This verification command also uses regex
grep. Using fixed-string matching avoids accidental matches and makes it clearer when nothing should be returned.
kubectl get manifestwork -n local-cluster | grep "${CLUSTER_ID}"
docs/ops/cleanup-stuck-cluster-deletion.md:515
- This section refers to
oc exec, but the rest of the runbook examples usekubectl exec. Consider aligning the wording to reduce cognitive overhead during an incident.
Port-forward is more reliable than `oc exec` for Maestro API inspection. It avoids response truncation and ensures all JSON fields (including root-level `deleted_at`) are visible.
| # TIP: Use port-forward instead of oc exec for more reliable Maestro API | ||
| # inspection -- oc exec can truncate responses or drop fields: | ||
| # kubectl port-forward -n maestro svc/maestro 8002:8000 | ||
| # curl -s 'http://localhost:8002/api/maestro/v1/resource-bundles?size=2900' | ... |
| jq '[.items[] | {id, name, deleted_at}]' | ||
| ``` | ||
|
|
||
| Use port-forward (`kubectl port-forward -n maestro svc/maestro 8002:8000`) instead of `oc exec` for more reliable API inspection — `oc exec` can truncate responses or drop fields. |
| kubectl wait --for=delete pod -l app=maestro-agent -n maestro --timeout=60s | ||
|
|
||
| # 3. Patch finalizers and delete all ManifestWorks for the stuck clusters | ||
| for mw in $(kubectl get manifestwork -n local-cluster -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | grep "${CLUSTER_ID}"); do |
- Use kubectl exec consistently instead of oc exec - Use grep -F for fixed-string matching on cluster IDs
|
|
||
| ```bash | ||
| # 1. Scale down the Maestro Agent to stop ManifestWork recreation | ||
| kubectl scale deployment -n maestro maestro-agent --replicas=0 |
There was a problem hiding this comment.
The prerequisite should be: verify that the Maestro Agent scale-down will not affect ANY customer operations — not just existing clusters, but also any in-flight or incoming create/update/delete requests targeting this management cluster. While the Agent is down, any ARM operation that requires a ManifestWork to be applied on this MC will fail or time out. The safest precondition is confirming the MC has no customer-facing traffic (no customer clusters AND the region is not actively serving new requests). If the region is live with customers, this procedure requires a maintenance window.
There was a problem hiding this comment.
Good call — added a prerequisite warning block in 50ae3fb covering the full blast radius (all ManifestWork delivery blocked, in-flight ARM ops fail/timeout, need to confirm no customer traffic or take a maintenance window). Also added a fallback reference to Strategy 4 (manual finalizer removal) for cases where a maintenance window isn't available, since it patches individual resources without stopping the agent.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
docs/ops/cleanup-stuck-cluster-deletion.md:318
- The prerequisite text suggests using Strategy 4 if a maintenance window isn’t available, but Strategy 4 explicitly requires that the source will not recreate resources (and Scenario 7 is defined by the agent recreating ManifestWorks). This is internally inconsistent and could mislead operators into a cleanup that won’t stick; consider directing readers to stop and schedule a maintenance window (or evacuate clusters) if they cannot scale the agent to 0.
> If the region is live with customers, this procedure **requires a maintenance window**.
>
> If a maintenance window is not available, use [Strategy 4 (Manual Finalizer Removal)](#strategy-4-manual-finalizer-removal-on-management-cluster-last-resort) instead. Strategy 4 patches finalizers on individual resources without stopping the agent, so it does not block other operations on the MC.
docs/ops/cleanup-stuck-cluster-deletion.md:349
- Strategy 3 scales the maestro-agent back up with
--replicas=1, but the deployment’s replica count is environment-configurable (seemaestro/agent/values.yamlusing.maestro.agent.k8s.replicas). Restoring to 1 could unintentionally reduce availability; capture the original replica count before scaling down and restore to that value.
# 5. Scale the Maestro Agent back up
kubectl scale deployment -n maestro maestro-agent --replicas=1
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: avollmer-redhat, weherdh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Why
During AROSLSRE-1520 (50 HCP clusters stuck deleting in brazilsouth after a scale test), we discovered a new Maestro failure mode that isn't covered by the existing runbook scenarios. The Maestro Agent continued recreating ManifestWorks even though the ResourceBundles were already soft-deleted on the server side — a CloudEvents sync desync caused by ARO-28432. Resolution required scaling the agent down (not just restarting) to eliminate a race condition during cleanup.
We also hit a significant debugging delay because the
deleted_atfield lives at the root level of the Maestro ResourceBundle JSON, not inside.metadata— existingjqqueries were checking the wrong path and showing bundles as "clean" when they weren't.What
docs/ops/cleanup-stuck-cluster-deletion.mddeleted_atis at the JSON root level, not.metadata.deleted_at— incorrect queries caused hours of debugging delaykubectl port-forward -n maestro svc/maestro 8002:8000is more reliable thanoc execfor API inspection.metadata.deleted_atdocs/ops/fix-maestro-stale-resource-bundle.mddeleted_atfield location warningReferences