Improve snapshot analyze context for stuck deletion scenario - #5989
Improve snapshot analyze context for stuck deletion scenario#5989venkateshsredhat wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR enhances hcpctl snapshot analyze troubleshooting guidance and data collection for cluster deletions that stall during the Clusters Service (CS) sequential destruct chain—specifically by surfacing mgmt-agent PodWatcher pod lifecycle/eviction signals and documenting how addon pre-delete pod eviction can block ManagedCluster detaching.
Changes:
- Added two new mgmt-agent snapshot KQL queries (
podEvents,podEvictions) for PodWatcher-derived pod lifecycle + eviction visibility inklusterlet-<cluster-id>. - Extended snapshot query README guidance and agent reference docs to describe the CS destruct chain and how it can block at the ManagedCluster/addon layer.
- Added a worked RCA exemplar for a “deletion stuck due to addon pre-delete pod eviction (MemoryPressure)” scenario and linked it from the AI query cookbook.
PR standards notes (per CONTRIBUTING.md#pull-request-standards):
- The PR title does not appear to follow Conventional Commits format.
- The PR description excerpt does not include a tracking ticket/issue reference.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvictions/README.md | Documents how to interpret eviction-focused PodWatcher results for klusterlet namespace pods. |
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvictions/query.kql | Adds KQL to filter PodWatcher events down to evicted pods for the cluster’s klusterlet namespace. |
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvents/README.md | Documents what pod lifecycle signals to look for (evictions, failures, deletes). |
| tooling/hcpctl/pkg/snapshot/queries/mgmtAgent/podEvents/query.kql | Adds KQL to list all PodWatcher pod events for the cluster’s klusterlet namespace. |
| tooling/hcpctl/pkg/snapshot/queries/clustersService/phases/README.md | Adds “uninstalling stuck” guidance pointing to destructor logs + pod eviction checks. |
| tooling/hcpctl/pkg/snapshot/queries/clustersService/logs/README.md | Adds cross-links from CS logs to mgmt-agent pod event/eviction queries when destruct chain stalls. |
| tooling/hcpctl/pkg/snapshot/queries.go | Registers the two new mgmt-agent queries in the snapshot query catalog. |
| tooling/hcpctl/pkg/agent/prompts/references/service-components.md | Adds ACM/ManagedCluster layer responsibilities, failure modes, and relevant logs for deletion analysis. |
| tooling/hcpctl/pkg/agent/prompts/references/architecture.md | Documents the sequential CS destruct chain and where it can block during deletion. |
| tooling/hcpctl/pkg/agent/prompts/exemplars/cluster-deletion-addon-pod-eviction.md | Adds an end-to-end RCA exemplar for addon pre-delete pod eviction blocking deletion. |
| docs/ai/query-cookbook.md | Links the new exemplar from the cookbook’s worked-examples section. |
|
/hold |
92091cd to
2589687
Compare
| @@ -21,6 +21,31 @@ Customer → ARM → Frontend → Backend (async) → Clusters Service → Maest | |||
| 5. **HyperShift** reconciles HostedCluster and NodePool custom resources on the management cluster, | |||
| creating the actual control plane pods in a hosted-control-plane namespace. | |||
|
|
|||
| ## Deletion Flow | |||
2589687 to
bf79d6c
Compare
| ```kql | ||
| clustersServiceLogs | ||
| | where timestamp between (datetime(2026-07-01) .. datetime(2026-07-03)) | ||
| | where log has '2r9nhugpbdko2vai55lv2ikki9h9958r' |
There was a problem hiding this comment.
maybe we can indicate how to retrieve this from the manifest.json;
| | Running destructor 'break-glass-credential-secrets-deleter' for cluster | 7/2/2026, 10:13:38.805 AM | 7/2/2026, 10:19:17.932 AM | 64 | | ||
| | Running destructor 'swift-podnetworkinstance-deleter' for cluster | 7/2/2026, 10:13:38.822 AM | 7/2/2026, 10:19:17.938 AM | 64 | | ||
|
|
||
| ### Why was the `ManagedCluster` destructor stuck? |
There was a problem hiding this comment.
We'll be better off first check the ManagedCluster CR conditions;
As written here it feels like a step was skipped;
The condition will tell that the ManagedCluster CR is awaiting for something to be deleted;
Then check why that something is not finalised;
which lands to the uninstall pods not running
There was a problem hiding this comment.
you also likely need to cite the cs codebase itself to tell us what Running destructor 'hypershift-managed-cluster-destructor' for cluster does - cite the code to make a normative claim of "what ought to happen"
There was a problem hiding this comment.
yes done .
| containerLogs | ||
| | where timestamp between (datetime(2026-07-01) .. datetime(2026-07-03)) | ||
| | where namespace_name == "mgmt-agent" and log.msg == "pod event" | ||
| | where log.namespace == "klusterlet-2r9nhugpbdko2vai55lv2ikki9h9958r" |
There was a problem hiding this comment.
give a hint on how to retrieve the ocm cluster id as indicated above
There was a problem hiding this comment.
we might need a new set of pre-canned queries for events & logs from components we care about in the klusterlet- namespace
| **Suggestions:** | ||
|
|
||
| - The addon pre-delete hook mechanism should be resilient to transient pod eviction. Consider using a `Job` with | ||
| `restartPolicy: OnFailure` and a configurable `backoffLimit` so that evicted pods are automatically retried. | ||
| - Addon pre-delete pods could use `PriorityClass` or `PodDisruptionBudget` settings to reduce the likelihood of | ||
| eviction during critical cleanup operations. | ||
| - The destruct chain in Clusters Service could implement a timeout after which it force-removes addon finalizers | ||
| and proceeds with the rest of the chain, preventing indefinite blocking. |
|
|
||
| 1. **ARM** delivers the DELETE request to the **Frontend**, which creates an async operation. | ||
| 2. The **Backend** translates it into a Clusters Service API call. | ||
| 3. **Clusters Service** sets the cluster state to `'uninstalling'` and runs the destruct chain: |
There was a problem hiding this comment.
There are more deletion chains; and node pool has different set.
I wonder if we can make it generic?
There was a problem hiding this comment.
I suggest to remove the mgmtAgent folder in favor of an acm folder in https://github.com/Azure/ARO-HCP/tree/5a8758f3b797aaf54f3caee3d565a37bf29f2eb6/tooling/hcpctl/pkg/snapshot/queries
that contains managedClusterConditions folder and enlist all the managedCluster CR conditions: similar to how its done for the hostedCluster or node pool but leveraging the logs of the CR from the mgmt-agent;
The conditions will indicate why deletion is stuck.
This way we can drop https://github.com/Azure/ARO-HCP/tree/5a8758f3b797aaf54f3caee3d565a37bf29f2eb6/tooling/hcpctl/pkg/snapshot/queries/mgmtAgent which are kind of specific and are already captured in the examplar .
Once you've the managedCluster conditions;
Add a reference to them in: https://github.com/Azure/ARO-HCP/blob/5a8758f3b797aaf54f3caee3d565a37bf29f2eb6/tooling/hcpctl/pkg/snapshot/queries/clustersService/phases/README.md in place of what you've now.
cluster reaches uninstalling but never completes, review logs/clustersService/logs.md for repeated Not continuing to the next destructor messages, and check conditions/acm/managedClusterConditions.md for .....
PS: Notice how I used cluster, you might want to review the node pool destructor code as well to capture corresponding destructor logs and add a similar line
| | 7/2/2026, 2:13:58.836 AM | klusterlet-addon-workmgr-54b77c6b77-48rqm | Update | Evicted | The node was low on resource: memory. Threshold quantity: 100Mi, available: 35152Ki. | Failed | aks-infrasd4ds52-30241817-vmss000000 | | ||
|
|
||
| The `MemoryPressure` condition was transient — the node recovered later — but the addon pre-delete pods were | ||
| repeatedly evicted, permanently blocking the `ManagedCluster` cleanup until the cleaner intervened at ~10:12 AM. |
There was a problem hiding this comment.
Those pods look like they come from a Deployment + ReplicaSet or something. We see them being re-tried a few times - we need to dig in and understand why they are re-tried N times but not N+1. This doesn't match normal k8s semantics
- Remove hardcoded "Two" count from query-cookbook.md worked examples - Fix snapshot output path references to use conventions like logs/clustersService/logs.md and events/mgmtAgent/podEvictions.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
5a8758f to
d816f84
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
tooling/hcpctl/pkg/agent/prompts/references/architecture.md:65
- This reference looks like it intends to point at the snapshot output file, but it’s missing the .md extension (other snapshot references use *.md).
`conditions/acm/managedClusterConditions` for the ManagedCluster state when the chain is stuck.
tooling/hcpctl/pkg/snapshot/queries.go:651
- PR description mentions adding new snapshot queries named podEvents/podEvictions and an exemplar named cluster-deletion-addon-pod-eviction.md, but this PR adds managedClusterConditions/klusterletEvents and cluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md. Please update the PR description to match the actual changes, or add/rename the missing items.
// --- ACM: ManagedCluster conditions and klusterlet events ---
{
component: "acm",
queryName: "managedClusterConditions",
tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/query.kql:11
- The summarize/take_any here can pair an arbitrary ManagedCluster object with an arbitrary timestamp (because it groups by event and uses take_any). That makes the chosen conditions potentially not match the most recent ResourceWatcher emission in the window.
| summarize content=take_any(log.object), observedTime=take_any(timestamp) by event=tostring(log.event)
| top 1 by observedTime desc
| ### Node Pool Deletion | ||
|
|
||
| Node pool deletion follows a simpler path — there is no `uninstalling` phase and no dedicated | ||
| destruct chain. CS deletes the node pool's Maestro resource bundles directly, which cascades | ||
| through the ManifestWork to HyperShift for NodePool resource cleanup on the management cluster. |
There was a problem hiding this comment.
Node pool deletion follows a simpler path — there is no
uninstallingphase and no dedicated
This isn't true; Please see https://github.com/openshift-online/aro-hcp-clusters-service/blob/master/cmd/clusters-service/service/azure/aro_hcp_delete_node_pool_worker.go
The node pool deletion follows a similar flow as the cluster
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/query.kql:12
- The query uses
take_any(timestamp)and thentop 1 by observedTime, which can return a non-latest ManagedCluster object nondeterministically. To reliably snapshot the most recent ResourceWatcher emission, usearg_max(timestamp, log.object)and deriveobservedTimefrom the max timestamp.
| summarize content=take_any(log.object), observedTime=take_any(timestamp) by event=tostring(log.event)
| top 1 by observedTime desc
| mv-expand condition = content.status.conditions
tooling/hcpctl/pkg/agent/prompts/references/architecture.md:68
- This reference is missing the
.mdextension, unlike the other snapshot cross-references in this repo (e.g.conditions/acm/managedClusterConditions.md). As written, it’s likely to be treated as plain code text rather than a consistent link/identifier.
resolves or an operator intervenes. Check `conditions/acm/managedClusterConditions` for the
tooling/hcpctl/pkg/snapshot/queries.go:652
- PR description calls out adding two new mgmt-agent snapshot queries named
podEventsandpodEvictions, but this change addsmanagedClusterConditionsandklusterletEvents(and there are nopodEvents/podEvictionssnapshot queries present). Please align the PR description (or rename/adjust the queries) so reviewers/users can find the intended additions.
// --- ACM: ManagedCluster conditions and klusterlet events ---
{
component: "acm",
queryName: "managedClusterConditions",
templatePath: "queries/acm/managedClusterConditions/query.kql",
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/query.kql:13
- The query claims to extract conditions from the last mgmt-agent ResourceWatcher emission in the phase window, but
take_any(timestamp)makesobservedTimenondeterministic. As a result,top 1 by observedTime descmay not actually pick the most recent emission, and the conditions snapshot can be stale/incorrect.
| summarize content=take_any(log.object), observedTime=take_any(timestamp) by event=tostring(log.event)
| top 1 by observedTime desc
| mv-expand condition = content.status.conditions
| project observedTime, type=tostring(condition.type), status=tostring(condition.status), reason=tostring(condition.reason), message=tostring(condition.message), lastTransitionTime=todatetime(condition.lastTransitionTime)
tooling/hcpctl/pkg/snapshot/queries.go:652
- PR description says it adds two new mgmt-agent snapshot queries (podEvents, podEvictions) and an exemplar named
cluster-deletion-addon-pod-eviction.md, but this change set adds three queries (managedClusterConditions,klusterletEvents,klusterletLogs) and the exemplar file iscluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md. Consider updating the PR description (or renaming the queries/exemplar) so reviewers and future readers can reliably match the narrative to the implementation.
// --- ACM: ManagedCluster conditions and klusterlet events ---
{
component: "acm",
queryName: "managedClusterConditions",
templatePath: "queries/acm/managedClusterConditions/query.kql",
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/query.kql:13
- The query is intended to return the latest ManagedCluster condition snapshot, but
take_any()combined with grouping byeventcan decoupleobservedTimefrom the correspondinglog.object, andtop 1can pick an arbitrary event row. This can yield conditions from an older object while labeling it with a newer timestamp (or vice versa). Consider selecting the single most-recent row witharg_max(timestamp, ...)and then expanding conditions from that object.
| summarize content=take_any(log.object), observedTime=take_any(timestamp) by event=tostring(log.event)
| top 1 by observedTime desc
| mv-expand condition = content.status.conditions
| project observedTime, type=tostring(condition.type), status=tostring(condition.status), reason=tostring(condition.reason), message=tostring(condition.message), lastTransitionTime=todatetime(condition.lastTransitionTime)
docs/ai/query-cookbook.md:182
- PR description mentions new snapshot queries named
podEvents/podEvictionsand an exemplarcluster-deletion-addon-pod-eviction.md, but this PR addsacm/klusterletLogs,acm/klusterletEvents, andcluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md. Please align the PR description with the actual query/exemplar names to avoid confusion when searching docs/history.
- `cluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md` — cluster deletion stuck at ManagedCluster destructor due to addon pod eviction (MemoryPressure)
tooling/hcpctl/pkg/agent/prompts/references/architecture.md:52
- This section lists specific cluster destructors, but the exemplar in this PR shows additional destructors (e.g. namespaces / Azure cleanup) appearing in Clusters Service logs. As written, this reads like a complete list and could mislead analysis. Suggest marking the list as non-exhaustive and pointing readers to CS logs for the authoritative step list.
3. **Clusters Service** sets the cluster state to `'uninstalling'` and runs the cluster destruct
chain (see `aro-hcp-clusters-service` repo, `pkg/clusterprovisioner/acm/destruct/`):
- `hypershift-managed-cluster-destructor`: waits for the **ManagedCluster** (ACM/MCE) CR
to be fully deleted. Deletion requires **ManagedClusterAddon** pre-delete hook pods to
complete and remove their finalizers first. If the ManagedCluster CR still exists, the
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: machi1990, venkateshsredhat The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
for a cluster's klusterlet namespace
pre-delete pods were evicted under MemoryPressure
destructor model and where it can block