Improve snapshot analyze context for stuck deletion scenario

docs/ai/query-cookbook.md

@@ -171,14 +171,15 @@ interactive debugging session.
 
 ## Worked-example analyses
 
-Two fully-worked end-to-end analyses (causal chain, queries, code citations)
+End-to-end analyses (causal chain, queries, code citations)
 live in
 [tooling/hcpctl/pkg/agent/prompts/exemplars/](../../tooling/hcpctl/pkg/agent/prompts/exemplars/):
 
 - `cluster-installation-azure-disk-failure.md`
 - `cluster-cleanup-unknown-failure.md`
 - `kubernetes-events.md` — `ServiceLogs.kubernetesEvents` and ad-hoc filtering beyond snapshot `controlPlaneEvents`
 - `mgmt-agent-event-logs.md` — ad-hoc KQL for mgmt-agent `resource event` / `pod event` timelines
+- `cluster-deletion-timeout-because-managedcluster-cr-deletion-stuck.md` — cluster deletion stuck at ManagedCluster destructor due to addon pod eviction (MemoryPressure)
 
 Read these when you need a concrete template for what a thorough RCA looks
 like in this stack.

tooling/hcpctl/pkg/agent/prompts/references/architecture.md

@@ -25,6 +25,66 @@ Customer → ARM → Frontend → Backend (async) → Clusters Service → Maest
 7. **HyperShift** reconciles HostedCluster and NodePool custom resources on the management cluster,
    creating the actual control plane pods in a hosted-control-plane namespace.
 
+## Deletion Flow
+
+A customer deletion operation follows the same component chain, but cleanup on the management
+cluster involves a sequential **destruct chain** managed by Clusters Service.
+
+### Cluster Deletion
+
+```
+Customer → ARM → Frontend → Backend (async) → Clusters Service → Management Cluster cleanup
+```
+
+1. **ARM** delivers the DELETE request to the **Frontend**, which creates an async operation.
+2. The **Backend** translates it into a Clusters Service API call.
+3. **Clusters Service** sets the cluster state to `'uninstalling'` and runs the cluster destruct
+   chain (see `aro-hcp-clusters-service` repo, `pkg/clusterprovisioner/acm/destruct/`):
+   - `hypershift-managed-cluster-destructor`: waits for the **ManagedCluster** (ACM/MCE) CR
+     to be fully deleted. Deletion requires **ManagedClusterAddon** pre-delete hook pods to
+     complete and remove their finalizers first. If the ManagedCluster CR still exists, the
+     destructor returns without advancing.
+   - `hypershift-manifest-work-destructor`: deletes Maestro resource bundles, which removes
+     ManifestWork objects, cascading to HostedCluster / NodePool / control plane deletion.
+   - `break-glass-credential-secrets-deleter`: removes break-glass credential secrets.
+   - `swift-podnetworkinstance-deleter`: removes PodNetworkInstance resources.
+4. **HyperShift** reconciles the HostedCluster deletion, cleaning up the control plane namespace
+   and cloud resources.
+
+### Node Pool Deletion
+
+Node pool deletion follows a simpler path with its own 3-step destruct chain
+(see `aro-hcp-clusters-service` repo, `pkg/nodepoolprovisioner/acm/aro/destruct/`):
+
+1. The API handler sets the node pool state to `'uninstalling'`. Once in this state, the node
+   pool cannot transition to any other state.
+2. The delete node pool worker picks it up and runs the destruct chain:
+   - `nodePoolCrDestructor`: deletes the ManifestWork for the node pool from the service cluster.
+     ACM propagates the deletion to the management cluster, where HyperShift drains nodes and
+     deallocates Azure VMs. The destructor waits until the ManifestWork is fully gone.
+   - `RhManagedNsgSubnetDissociationDestructor`: removes the RH-managed NSG association from the
+     node pool's subnet (only if the subnet differs from the cluster default).
+   - `nodePoolDbDestructor`: deletes the node pool record and associated resource key from the DB.
+
+Unlike cluster deletion, node pool deletion does not involve ManagedCluster, addon pre-delete
+hooks, or namespace cleanup — it only needs the ManifestWork removal, Azure networking cleanup,
+and DB record deletion.
+
+### Destruct Chain Behavior
+
+Both cluster and node pool destruct chains are **sequential** and run as Go function calls within
+the Clusters Service process (not as Kubernetes Jobs on the management cluster). Each chain
+restarts from step 0 on each reconcile — completed steps return immediately since their resources
+are already gone. If one destructor cannot complete (e.g. ManagedCluster CR stuck pending
+deletion, or a ManifestWork still has a `DeletionTimestamp`), all subsequent destructors are
+skipped. CS logs `destructor ... has not completed yet` or `Not continuing to the next destructor`
+on each iteration. There is no built-in timeout — the chain retries indefinitely until the
+blocking step resolves or an operator intervenes.
+
+For cluster deletion stuck at the ManagedCluster step, check
+`conditions/acm/managedClusterConditions` for the ManagedCluster state. For node pool deletion
+stuck at the CR destructor step, check HyperShift NodePool conditions and ManifestWork status.
+
 ## Topology
 
 - **Management clusters**: Run HyperShift, Maestro agent, kube-applier, and hosted control planes for many customers.

tooling/hcpctl/pkg/agent/prompts/references/service-components.md

@@ -90,3 +90,41 @@ into actual control plane infrastructure.
 
 **Failure modes:** Condition degraded/progressing stuck, pod crash loops, etcd issues, RBAC errors,
 resource quota exceeded, image pull failures.
+
+## ACM / ManagedCluster Layer
+
+Advanced Cluster Management (ACM) and Multicluster Engine (MCE) components manage cluster
+registration and addon lifecycle on the management cluster.
+
+**Key objects:**
+- `ManagedCluster` (cluster-scoped): represents a registered cluster, named by the CS cluster ID.
+  Finalizer: `cluster.open-cluster-management.io/api-resource-cleanup`.
+- `ManagedClusterAddon` (namespaced under the cluster ID): represents an addon installed on the
+  managed cluster. Common addons: `config-policy-controller`, `governance-policy-framework`.
+  Finalizers: `hosting-manifests-cleanup`, `hosting-addon-pre-delete`.
+
+**Responsibilities:**
+- Cluster registration and status reporting
+- Addon lifecycle management (install, upgrade, pre-delete hooks)
+
+**Role in deletion:**
+The CS destruct chain runs `hypershift-managed-cluster-destructor`, which checks ManagedCluster
+status. The ManagedCluster enters `Detaching` state, triggering addon pre-delete hooks. Each
+addon's pre-delete hook pod must complete and remove its finalizers before the ManagedCluster
+can finish detaching. Until all addons are cleaned up, the destruct chain is blocked.
+
+**Failure modes:**
+- Addon pre-delete pod eviction: node resource pressure (MemoryPressure, DiskPressure) evicts
+  the pre-delete hook pods before completion, leaving finalizers in place.
+- Klusterlet auth loss: the klusterlet identity loses Azure AD authorization during Detaching,
+  preventing pre-delete hooks from executing.
+- ManagedCluster stuck Detaching: any addon cleanup failure leaves the ManagedCluster in
+  Detaching state indefinitely, blocking the entire deletion chain.
+
+**Key logs:**
+- Pod lifecycle (including eviction): `containerLogs` from `mgmt-agent` namespace —
+  `log.msg == 'pod event'` and `log.namespace == 'klusterlet-<cluster-id>'`. Evicted pods
+  have `log.object.status.reason == 'Evicted'` with `log.object.status.message` containing
+  the node condition (e.g. `"The node was low on resource: memory"`).
+- Destruct chain: `clustersServiceLogs` — filter for `log has '<cluster-id>'` and
+  `log has 'destructor'` to see which destructor is blocking and how many iterations.

tooling/hcpctl/pkg/snapshot/queries.go

@@ -645,6 +645,42 @@ var allQueries = []querySpec{
 		requiredWhen:  isClusterOrNodePool,
 	},
 
+	// --- ACM: ManagedCluster conditions and klusterlet events ---
+	{
+		component:    "acm",
+		queryName:    "managedClusterConditions",
+		templatePath: "queries/acm/managedClusterConditions/query.kql",
+		database:     "service",
+		category:     categoryConditions,
+		ready: func(d queryData) bool {
+			return d.ClusterID != "" && strings.EqualFold(d.ResourceType, "microsoft.redhatopenshift/hcpopenshiftclusters")
+		},
+		prerequisites: "ClusterID, ResourceType is cluster",
+		requiredWhen:  isClusterType,
+	},
+	{
+		component:    "acm",
+		queryName:    "klusterletEvents",
+		templatePath: "queries/acm/klusterletEvents/query.kql",
+		database:     "service",
+		category:     categoryResourceEvents,
+		ready: func(d queryData) bool {
+			return d.ClusterID != "" && strings.EqualFold(d.ResourceType, "microsoft.redhatopenshift/hcpopenshiftclusters")
+		},
+		prerequisites: "ClusterID, ResourceType is cluster",
+	},
+	{
+		component:    "acm",
+		queryName:    "klusterletLogs",
+		templatePath: "queries/acm/klusterletLogs/query.kql",
+		database:     "service",
+		category:     categoryLogs,
+		ready: func(d queryData) bool {
+			return d.ClusterID != "" && strings.EqualFold(d.ResourceType, "microsoft.redhatopenshift/hcpopenshiftclusters")
+		},
+		prerequisites: "ClusterID, ResourceType is cluster",
+	},
+
 	// --- Events: time-windowed, component-scoped ---
 	{
 		component:    "frontend",

tooling/hcpctl/pkg/snapshot/queries/acm/klusterletEvents/README.md

@@ -0,0 +1,22 @@
+# acm / klusterletEvents
+
+## Summary
+
+Lists Kubernetes API events from the klusterlet namespace (`klusterlet-<cluster-id>`) for this cluster.
+These are K8s Event objects ingested by the kube-events collector into `ServiceLogs.kubernetesEvents`.
+
+## What to Look For
+
+- `Evicted` reason on addon pre-delete pods — indicates node resource pressure (MemoryPressure,
+  DiskPressure) evicted the pods before completion, blocking ManagedCluster deletion.
+- `FailedScheduling` reason — the pod could not be placed on any node.
+- `FailedMount`, `BackOff`, `Unhealthy` reasons — container-level failures that prevent
+  addon hooks from completing.
+
+## Where to Go Next
+
+- If addon pre-delete pods are being evicted, check `logs/acm/klusterletLogs.md` for pod-level detail
+  (status reason, phase, node placement) and `conditions/acm/managedClusterConditions.md` to confirm
+  the ManagedCluster is stuck pending deletion.
+- Review `logs/clustersService/logs.md` for the destruct chain state.
+- See `docs/ops/cleanup-stuck-cluster-deletion.md`, Scenario 6, for manual remediation.

tooling/hcpctl/pkg/snapshot/queries/acm/klusterletEvents/query.kql

@@ -0,0 +1,14 @@
+cluster('{{ .ClusterURI }}').database('{{ .ServiceDatabase }}').table('kubernetesEvents')
+| where timestamp between ({{ kqlDatetime .PhaseStartTime }} .. {{ kqlDatetime .PhaseEndTime }})
+{{- if and .ServiceClusterName .ManagementClusterName }}
+| where cluster in ('{{ .ServiceClusterName }}', '{{ .ManagementClusterName }}')
+{{- end }}
+| where eventNamespace == strcat('klusterlet-', '{{ .ClusterID }}')
+| project
+    timestamp,
+    objectKind,
+    objectName,
+    reason,
+    message,
+    type
+| order by timestamp asc

tooling/hcpctl/pkg/snapshot/queries/acm/klusterletLogs/README.md

@@ -0,0 +1,24 @@
+# acm / klusterletLogs
+
+## Summary
+
+Lists pod lifecycle events from the klusterlet namespace (`klusterlet-<cluster-id>`) as observed by the
+mgmt-agent PodWatcher. These are container log entries (not K8s Event objects) from the `mgmt-agent`
+namespace where `log.msg == 'pod event'`, providing pod-level detail including status reason, phase,
+and node placement.
+
+## What to Look For
+
+- Pods with `reason == 'Evicted'` and messages mentioning `MemoryPressure` or `DiskPressure` —
+  indicates node resource pressure evicted addon pods before they could complete.
+- `phase == 'Failed'` on addon pre-delete hook pods — confirms the pod did not run to completion.
+- Multiple eviction entries for the same ReplicaSet hash — suggests repeated reschedule-then-evict
+  cycles that exhaust the controller's retry budget.
+- Evictions across different nodes — indicates cluster-wide resource pressure, not a single-node issue.
+
+## Where to Go Next
+
+- Check `events/acm/klusterletEvents.md` for the corresponding K8s Event objects (Evicted, FailedScheduling).
+- Check `conditions/acm/managedClusterConditions.md` to see if the ManagedCluster is stuck pending deletion.
+- Review `logs/clustersService/logs.md` for the destruct chain state.
+- See `docs/ops/cleanup-stuck-cluster-deletion.md`, Scenario 6, for manual remediation.

tooling/hcpctl/pkg/snapshot/queries/acm/klusterletLogs/query.kql

@@ -0,0 +1,16 @@
+cluster('{{ .ClusterURI }}').database('{{ .ServiceDatabase }}').table('containerLogs')
+| where timestamp between ({{ kqlDatetime .PhaseStartTime }} .. {{ kqlDatetime .PhaseEndTime }})
+{{- if and .ServiceClusterName .ManagementClusterName }}
+| where cluster in ('{{ .ServiceClusterName }}', '{{ .ManagementClusterName }}')
+{{- end }}
+| where namespace_name == 'mgmt-agent' and log.msg == 'pod event'
+| where log.namespace == strcat('klusterlet-', '{{ .ClusterID }}')
+| project
+    timestamp,
+    pod_name = tostring(log.name),
+    event = tostring(log.event),
+    reason = tostring(log.object.status.reason),
+    message = tostring(log.object.status.message),
+    phase = tostring(log.object.status.phase),
+    node = tostring(log.object.spec.nodeName)
+| order by timestamp asc

tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/README.md

@@ -0,0 +1,33 @@
+# acm / managedClusterConditions
+
+## Summary
+
+Extracts a point-in-time snapshot of ManagedCluster conditions from the last mgmt-agent ResourceWatcher
+emission within the current phase's time window. The ManagedCluster CR is cluster-scoped and named by the
+CS cluster ID.
+
+## What to Look For
+
+A healthy ManagedCluster should have:
+
+| type                    | status | reason              |
+|-------------------------|--------|---------------------|
+| HubAcceptedManagedCluster | True | HubClusterAdminAccepted |
+| ManagedClusterConditionAvailable | True | ManagedClusterAvailable |
+| ManagedClusterJoined    | True   | ManagedClusterJoined |
+
+During deletion, watch for:
+
+- `ManagedClusterConditionAvailable` changing to `Unknown` — indicates the klusterlet has lost contact,
+  which can prevent addon pre-delete hooks from executing.
+- `ManagedClusterImportSucceeded` showing reason `ManagedClusterForceDetaching` — the hub is attempting
+  a forced detach, indicating the ManagedCluster is stuck pending deletion.
+
+## Where to Go Next
+
+- If conditions show the cluster stuck pending deletion, check `logs/clustersService/logs.md` for repeated
+  `Not continuing to the next destructor` messages at the `hypershift-managed-cluster-destructor` step.
+- Review `events/acm/klusterletEvents.md` for K8s events and `logs/acm/klusterletLogs.md` for pod-level
+  detail on evictions or scheduling failures in the klusterlet namespace.
+- See `docs/ops/cleanup-stuck-cluster-deletion.md`, Scenario 6, for the manual fix (clearing
+  `ManagedClusterAddon` finalizers).

tooling/hcpctl/pkg/snapshot/queries/acm/managedClusterConditions/query.kql

@@ -0,0 +1,13 @@
+cluster('{{ .ClusterURI }}').database('{{ .ServiceDatabase }}').table('containerLogs')
+| where timestamp between ({{ kqlDatetime .PhaseStartTime }} .. {{ kqlDatetime .PhaseEndTime }})
+{{- if and .ServiceClusterName .ManagementClusterName }}
+| where cluster in ('{{ .ServiceClusterName }}', '{{ .ManagementClusterName }}')
+{{- end }}
+| where container_name == 'mgmt-agent-controller'
+| where tostring(log.msg) == 'resource event'
+| where tostring(log.object.kind) == 'ManagedCluster'
+| where tostring(log.name) == '{{ .ClusterID }}'
+| summarize content=take_any(log.object), observedTime=take_any(timestamp) by event=tostring(log.event)
+| top 1 by observedTime desc
+| mv-expand condition = content.status.conditions
+| project observedTime, type=tostring(condition.type), status=tostring(condition.status), reason=tostring(condition.reason), message=tostring(condition.message), lastTransitionTime=todatetime(condition.lastTransitionTime)

tooling/hcpctl/pkg/snapshot/queries/clustersService/logs/README.md

@@ -10,4 +10,5 @@ Generally verbose logs, but may exhibit repeated errors when Azure infrastructur
 
 ## Where to Go Next
 
-Review `events/clustersService/events.md` to confirm that Clusters Service is functioning properly.
+- Review `events/clustersService/events.md` to confirm that Clusters Service is functioning properly.
+- If logs show the destruct chain stuck at a destructor (repeated `Not continuing to the next destructor`), check `conditions/acm/managedClusterConditions.md` for ManagedCluster status, `events/acm/klusterletEvents.md` for K8s events, and `logs/acm/klusterletLogs.md` for addon pod evictions or scheduling failures in the klusterlet namespace.

tooling/hcpctl/pkg/snapshot/queries/clustersService/phases/README.md

@@ -17,13 +17,14 @@ uninstalling:
 | 2026-05-15T08:25:38.041Z | updating cluster 'cid' state to 'ready'          |
 | 2026-05-15T08:40:32.633Z | updating cluster 'cid' state to 'uninstalling'   |
 
-Node pools do something similar, without an uninstalling phase:
+Node pools follow a similar pattern, and also transition to `uninstalling` during deletion:
 
 | timestamp                | msg                                                                           |
 |--------------------------|-------------------------------------------------------------------------------|
 | 2026-05-15T08:28:47.214Z | Node pool 'np' created for cluster 'cid' with state 'validating'              |
 | 2026-05-15T08:28:52.068Z | Node pool 'np' for cluster 'cid' state updated from 'pending' to 'installing' |
 | 2026-05-15T08:37:43.07Z  | Node pool 'np' for cluster 'cid' state updated from 'installing' to 'ready'   |
+| 2026-05-15T08:40:33.500Z | Node pool 'np' for cluster 'cid' state updated from 'ready' to 'uninstalling' |
 
 ## Where to Go Next
 
@@ -32,3 +33,7 @@ If the cluster or node pool:
 - reaches `pending` but not `installing`, review `logs/clustersService/provisionSteps.md` to see which provision step is stuck or failing.
 - reaches `installing` but not `ready`, review `logs/clustersService/logs.md` paying attention to timestamps, and review `conditions/hypershift/hostedClusterConditions.md` or `conditions/hypershift/nodePoolConditions.md` for the next layer of the stack.
 - reaches `ready` in Clusters Service but the ARM create operation stays `Provisioning`, review `conditions/backend/resourceControllerConditions.md` and `conditions/hypershift/hostedClusterConditions.md`.
+
+If the **cluster** reaches `uninstalling` but never completes, review `logs/clustersService/logs.md` for repeated `Not continuing to the next destructor` messages, and check `conditions/acm/managedClusterConditions.md` for the ManagedCluster stuck pending deletion. Review `events/acm/klusterletEvents.md` for K8s events and `logs/acm/klusterletLogs.md` for addon pre-delete pod evictions or scheduling failures that may be blocking the destruct chain.
+
+If a **node pool** reaches `uninstalling` but never completes, review `logs/clustersService/logs.md` for repeated `destructor ... has not completed yet` messages. The node pool destruct chain has 3 steps: ManifestWork deletion (waiting for HyperShift to clean up the NodePool CR), NSG subnet dissociation, and DB cleanup. If stuck at the ManifestWork step, check `conditions/hypershift/nodePoolConditions.md` for the NodePool CR state on the management cluster.