Skip to content

fix(dev-ci): upgrade opstool AKS node pools, not just control plane (AROSLSRE-1414) - #5988

Merged
openshift-merge-bot[bot] merged 2 commits into
Azure:mainfrom
raelga:rael/aroslsre-1414-opstool-nodepool-upgrade
Jul 9, 2026
Merged

fix(dev-ci): upgrade opstool AKS node pools, not just control plane (AROSLSRE-1414)#5988
openshift-merge-bot[bot] merged 2 commits into
Azure:mainfrom
raelga:rael/aroslsre-1414-opstool-nodepool-upgrade

Conversation

@raelga

@raelga raelga commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

AROSLSRE-1414

What

Adds an upgrade-aks-cluster Shell step to the opstool dev-ci pipeline so that a Kubernetes version bump in config-dev-ci.yaml rolls the node pools, not just the control plane. The step runs the same dev-infrastructure/scripts/upgrade-aks-cluster.sh used by the svc/mgmt pipelines and assumes the shared global-rollout-identity, exposed via a new global-identity output step (mirroring the dev-ci e2e-subscription-rbac pipeline).

Why

The opstool pipeline previously ran only the cluster ARM step. That step sets the managedCluster kubernetesVersion (control plane) but the agentPoolProfiles / aks/pool.bicep pools carry no orchestratorVersion — on a managedCluster PUT an omitted orchestratorVersion means "no change", so existing node pools stay on their current version.

Observed after deploying the 1.33 bump to opstool: control plane went to 1.33.12 but all three node pools stayed on 1.32 orchestrator. svc/mgmt don't hit this because their pipelines already run upgrade-aks-cluster.sh (az aks upgrade, which rolls control plane and node pools). opstool was the outlier. This is also the root cause of the opstool version drift behind AROSLSRE-863.

config kubernetesVersion bump
  -> cluster ARM step        => control plane upgraded, pools unchanged  (before)
  -> upgrade-aks-cluster.sh  => `az aks upgrade` rolls control plane + node pools  (after)

Testing

  • templatize pipeline validate --topology-config-file topology-dev-ci.yaml --service-config-file config/config-dev-ci.yaml --dev-mode --dev-region westus3 → passes.
  • yamlfmt → no changes.
  • Reuses the already-proven upgrade-aks-cluster.sh and the global-identity output pattern from the sibling e2e-subscription-rbac pipeline; the script is idempotent (no-op when the cluster is already at the target version).

Special notes for your reviewer

dev-ci infra is rolled out manually (make dev-ci-local-run), so this takes effect on the next opstool rollout. Scope is dev-ci only (topology-dev-ci.yaml); int/stg/prod are unaffected.

…AROSLSRE-1414)

The opstool dev-ci pipeline only runs the 'cluster' ARM step, which sets
the managedCluster kubernetesVersion (control plane) but leaves existing
node pools on their current orchestratorVersion. As a result a Kubernetes
version bump in config-dev-ci.yaml upgraded only the opstool control
plane, leaving the node pools a minor behind.

Add an 'upgrade-aks-cluster' Shell step (running the same
upgrade-aks-cluster.sh used by svc/mgmt) so the node pools are rolled to
the configured version via 'az aks upgrade'. It assumes the shared
global-rollout-identity, exposed through a new 'global-identity' output
step mirroring the dev-ci e2e-subscription-rbac pipeline.
Copilot AI review requested due to automatic review settings July 9, 2026 08:17
@openshift-ci openshift-ci Bot added the approved label Jul 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the dev-ci opstool AKS rollout pipeline so that when opstool.aks.kubernetesVersion is bumped, the pipeline runs the existing upgrade-aks-cluster.sh flow (via az aks upgrade) to roll node pools as well as the control plane, aligning opstool behavior with svc/mgmt pipelines.

Changes:

  • Add an upgrade-aks-cluster Shell step that runs dev-infrastructure/scripts/upgrade-aks-cluster.sh with CLUSTER_NAME, RESOURCE_GROUP, and KUBERNETES_VERSION from config.
  • Add a new global resource group section that exposes the shared rollout identity via an output-only global-identity ARM step (used by the Shell step’s shellIdentity).

Comment thread dev-infrastructure/dev-ci/cluster/opstool-aks-pipeline.yaml
…LSRE-1414)

Repoint cluster-output and alerting dependsOn from 'cluster' to
'upgrade-aks-cluster' so the output reads, kusto-placeholder and the
prometheus Helm deploy only run after 'az aks upgrade' finishes rolling
the node pools, matching the svc/mgmt ordering where all post-cluster
steps depend on upgrade-aks-cluster.
Copilot AI review requested due to automatic review settings July 9, 2026 10:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@inbharajmani

Copy link
Copy Markdown
Collaborator

/lgtm

@openshift-ci

openshift-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inbharajmani, raelga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@raelga

raelga commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

Retesting e2e-parallel — this is a flake unrelated to the PR.

Failure: the single failing test is test/e2e/nodepool_version_upgrade.go:210failed to upgrade node pool npupgrade-4-21 to version 4.21.0 ... timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait ... context deadline exceeded. That's a timeout-class flake (known shared-CI-subscription ARM slowness), not a deterministic failure.

Scope evidence (not PR-specific):

/test e2e-parallel

@openshift-merge-bot
openshift-merge-bot Bot merged commit 1ba50ff into Azure:main Jul 9, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants