fix(dev-ci): upgrade opstool AKS node pools, not just control plane (AROSLSRE-1414) - #5988
Conversation
…AROSLSRE-1414) The opstool dev-ci pipeline only runs the 'cluster' ARM step, which sets the managedCluster kubernetesVersion (control plane) but leaves existing node pools on their current orchestratorVersion. As a result a Kubernetes version bump in config-dev-ci.yaml upgraded only the opstool control plane, leaving the node pools a minor behind. Add an 'upgrade-aks-cluster' Shell step (running the same upgrade-aks-cluster.sh used by svc/mgmt) so the node pools are rolled to the configured version via 'az aks upgrade'. It assumes the shared global-rollout-identity, exposed through a new 'global-identity' output step mirroring the dev-ci e2e-subscription-rbac pipeline.
There was a problem hiding this comment.
Pull request overview
Updates the dev-ci opstool AKS rollout pipeline so that when opstool.aks.kubernetesVersion is bumped, the pipeline runs the existing upgrade-aks-cluster.sh flow (via az aks upgrade) to roll node pools as well as the control plane, aligning opstool behavior with svc/mgmt pipelines.
Changes:
- Add an
upgrade-aks-clusterShell step that runsdev-infrastructure/scripts/upgrade-aks-cluster.shwithCLUSTER_NAME,RESOURCE_GROUP, andKUBERNETES_VERSIONfrom config. - Add a new
globalresource group section that exposes the shared rollout identity via an output-onlyglobal-identityARM step (used by the Shell step’sshellIdentity).
…LSRE-1414) Repoint cluster-output and alerting dependsOn from 'cluster' to 'upgrade-aks-cluster' so the output reads, kusto-placeholder and the prometheus Helm deploy only run after 'az aks upgrade' finishes rolling the node pools, matching the svc/mgmt ordering where all post-cluster steps depend on upgrade-aks-cluster.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: inbharajmani, raelga The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Retesting Failure: the single failing test is Scope evidence (not PR-specific):
/test e2e-parallel |
AROSLSRE-1414
What
Adds an
upgrade-aks-clusterShell step to the opstool dev-ci pipeline so that a Kubernetes version bump inconfig-dev-ci.yamlrolls the node pools, not just the control plane. The step runs the samedev-infrastructure/scripts/upgrade-aks-cluster.shused by the svc/mgmt pipelines and assumes the sharedglobal-rollout-identity, exposed via a newglobal-identityoutput step (mirroring the dev-cie2e-subscription-rbacpipeline).Why
The opstool pipeline previously ran only the
clusterARM step. That step sets the managedClusterkubernetesVersion(control plane) but theagentPoolProfiles/aks/pool.biceppools carry noorchestratorVersion— on a managedCluster PUT an omittedorchestratorVersionmeans "no change", so existing node pools stay on their current version.Observed after deploying the
1.33bump to opstool: control plane went to1.33.12but all three node pools stayed on1.32orchestrator. svc/mgmt don't hit this because their pipelines already runupgrade-aks-cluster.sh(az aks upgrade, which rolls control plane and node pools). opstool was the outlier. This is also the root cause of the opstool version drift behind AROSLSRE-863.Testing
templatize pipeline validate --topology-config-file topology-dev-ci.yaml --service-config-file config/config-dev-ci.yaml --dev-mode --dev-region westus3→ passes.yamlfmt→ no changes.upgrade-aks-cluster.shand theglobal-identityoutput pattern from the siblinge2e-subscription-rbacpipeline; the script is idempotent (no-op when the cluster is already at the target version).Special notes for your reviewer
dev-ci infra is rolled out manually (
make dev-ci-local-run), so this takes effect on the next opstool rollout. Scope is dev-ci only (topology-dev-ci.yaml); int/stg/prod are unaffected.