test: add upgrade/in-place E2E suite with node pool stability validation by patriksuba · Pull Request #5725 · Azure/ARO-HCP

Patrik Suba (patriksuba) · 2026-06-19T10:04:29Z

What

Adds an upgrade/in-place E2E suite that exercises the full regional in-place upgrade flow end-to-end. Multiple specs run in parallel — each provisions its own HCP cluster and node pool and captures pre-upgrade baselines — then all specs synchronise at an UpgradeBarrier while the UpgradeCoordinator (running in the long-lived parent run-suite process) invokes the Region entrypoint pipeline once for the whole suite via templatize's run.RunPipeline. After the upgrade every spec validates its own cluster independently.

New files

test/e2e/region_upgrade.go — provisions an HCP cluster and node pool, captures three pre-upgrade baselines:
- SHA-256 hash of node identity state (Name, UID, KubeletVersion, OSImage)
- HAProxy image reference (from k8s-app=kube-apiserver-proxy pods)
- MachineDeployment DataSecretName (MCO raw-config hash)
After CheckIn, specs perform during-upgrade validation using Consistently(...).WithContext(upgradeDoneCtx).WithPolling(...) — the observation window runs until the coordinator signals completion, not for a fixed duration. After WaitForUpgrade, specs assert all three baselines are unchanged post-upgrade.
test/util/framework/upgrade_coordinator.go — UpgradeCoordinator runs in the parent run-suite process (guarded by isRunSuiteProcess() in BeforeAll). It initialises the shared state file with the parent PID as run_id, polls for all specs to check in (waitSettled), then calls run.RunPipeline (Region entrypoint) in-process. A named-return defer markUpgradeDone(runErr) ensures the completion signal is written on every exit path — success, failure, env-var misconfiguration, or context cancellation — so no checked-in waiter ever hangs. The coordinator logger is injected into the pipeline context via logr.NewContext so runRegionEntrypoint logs are visible in CI's build-log.txt.
test/util/framework/upgrade_barrier.go — UpgradeBarrier coordinates parallel worker specs across OS-process boundaries using a YAML state file protected by syscall.Flock and atomic rename. CheckIn atomically increments checked_in and returns immediately with an upgradeDoneCtx (cancelled when the coordinator signals completion), allowing specs to start during-upgrade validation without waiting for peers. WaitForUpgrade polls the state file until UpgradeDone is set. CheckInAndWait is a convenience wrapper for specs that skip during-upgrade validation. Per-phase timeouts (settleTimeout=45min, upgradeRunTimeout=50min coordinator budget, upgradeTimeout=60min spec safety-net) prevent specs from hanging until Prow's hard job kill.

Stale-state detection uses run_id = os.Getppid() (parent PID): all parallel workers in the same suite invocation share the same parent PID; a new invocation gets a different one and resets any leftover state file. The spec count is in-memory only (b.total), populated at runtime by SetUpgradeInPlaceSpecCount — no env var or constant needs updating when specs are added or removed.
test/util/framework/upgrade_barrier_test.go — unit tests covering check-in/abort, all-abort early exit, coordinator-driven upgrade signalling, stale state detection, idempotent markUpgradeDone, context cancellation, and the all-aborted path writing UpgradeDone (harmless, no waiter polls).

Modified files

test/util/labels/labels.go — added UpgradeInPlace label for suite filtering; excluded from all other suites.
test/cmd/aro-hcp-tests/main.go — registered upgrade/in-place suite (120-minute timeout). Counts UpgradeInPlace specs from the binary itself at startup and calls framework.SetUpgradeInPlaceSpecCount — suite parallelism and barrier total are always in sync without any constant or env var. A BeforeAll guard starts the UpgradeCoordinator goroutine in the parent process only.
test/AGENTS.md — documented the UpgradeCoordinator + UpgradeBarrier architecture, timing table, stale-state detection, and It-block skeleton for both CheckInAndWait and the split CheckIn + during-upgrade validation + WaitForUpgrade pattern.
Prow step (aro-hcp-test-local-upgrade-commands.sh) — removed UPGRADE_SPEC_COUNT export; the test binary computes the spec count internally.

Why

Upgrade E2E coverage currently stops at cluster provisioning. There is no automated signal that confirms nodes are stable after a full regional upgrade — unexpected node churn (VM replacements, unintended rolling updates triggered by MCO config hash rotation or image substitution) goes undetected until a customer reports an issue.

This PR introduces parallel upgrade validation. Each upgrade/in-place spec provisions its own baseline cluster, then all specs collectively trigger the Region entrypoint pipeline (upgrading both service and management cluster infrastructure and all services in one idempotent invocation) and independently verify that:

the SHA-256 fingerprint of every node's identity is unchanged (catches VMSS replacements even if the node name and OS version are preserved, because node.UID is included)
the HAProxy image reference is unchanged (catches unintended registry-override substitution)
the MachineDeployment DataSecretName is unchanged (catches MCO raw-config hash rotation that would trigger a node rollout)

The upgrade runs in the parent run-suite process (not inside a worker spec) so its lifetime is independent of any individual spec failure — a spec crashing during provisioning cannot abort the shared upgrade or leave other specs hanging.

Testing

Verified locally against a personal dev environment as described in openshift/release#80778.

Special notes for your reviewer

The UpgradeCoordinator relies on isRunSuiteProcess() to distinguish the parent run-suite process from worker run-test processes. Under openshift-tests-extension, run-test workers re-execute main() with a different subcommand, so the guard is: os.Args contains run-suite and the invocation targets the upgrade/in-place suite. This is admittedly a convention of the current extension model — a cleaner lifecycle hook in the extension framework would be preferable long-term.

PR Checklist

PR is scoped to a single task (no mixed concerns)
Title follows Conventional Commits format
Summary explains the "Why" behind the change
Linked to relevant ticket/issue
Self-reviewed the diff
CI/CD checks are passing (ignore Tide)
Commit history is clean (rebased/squashed)
Tricky code blocks are commented
All comment threads resolved before merge

openshift-ci · 2026-06-19T10:04:39Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Patrik Suba (patriksuba) · 2026-06-19T10:06:24Z

/label ai-assisted

openshift-ci · 2026-06-19T10:06:27Z

Patrik Suba (@patriksuba): The label(s) /label ai-assisted cannot be applied. These labels are supported: acknowledge-critical-fixes-only, platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, ux-approved, no-qe, rebase/manual, cluster-config-api-changed, run-integration-tests, verified, ready-for-human-review, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/skip-dependent-bug-check, jira/valid-bug, lgtm, ok-to-test, stability-fix-approved, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

Details

In response to this:

/label ai-assisted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull request overview

Adds a new dedicated E2E suite (upgrade/in-place) to validate HypershiftOperator in-place upgrades end-to-end, including a post-upgrade “no node churn” stability check, and introduces a small framework utility for running make targets with context-aware process-group shutdown.

Changes:

Introduces an upgrade/in-place E2E spec that snapshots node identity/version state and asserts stability for a fixed post-upgrade window.
Adds a reusable MakeRunner helper (plus unit tests) to run make targets with SIGTERM→SIGKILL escalation on context cancellation.
Registers the new suite in aro-hcp-tests and adds a new UpgradeInPlace label for suite filtering.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/util/labels/labels.go	Adds `UpgradeInPlace` label for selecting the new suite.
test/util/framework/make_runner.go	Adds `MakeRunner` for running `make` targets with process-group cleanup on cancellation.
test/util/framework/make_runner_test.go	Unit coverage for `MakeRunner` success/failure/cancellation/output/env/workdir behavior.
test/testdata/zz_fixture_TestMainListSuitesForEachSuite_upgrade_in_placeupgrade_in_place.txt	Updates fixture output for suite listing.
test/e2e/hypershiftoperator_upgrade.go	New upgrade/in-place E2E spec that hashes node state and checks stability post-upgrade.
test/cmd/aro-hcp-tests/main.go	Registers the `upgrade/in-place` suite (parallelism 1, 120-minute timeout).
test/cmd/aro-hcp-tests/main_test.go	Extends suite-listing test coverage to include `upgrade/in-place`.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Patrik Suba (patriksuba) · 2026-07-10T09:44:46Z

/test images

Patrik Suba (patriksuba) · 2026-07-10T10:00:14Z

/test images

Patrik Suba (patriksuba) · 2026-07-10T10:05:13Z

The new test was successfully verified locally against a personal dev environment configured as shown in this PR: openshift/release#80778

Patrik Suba (patriksuba) · 2026-07-10T12:37:44Z

/test images

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.

- Replace markAndReturn helper with a named-return defer in UpgradeCoordinator.Run so markUpgradeDone fires on every exit path, including readState failure; update test to assert UpgradeDone is written even when all specs aborted (harmless — no waiter is polling) - Inject coordinator logger into the pipeline context via logr.NewContext so runRegionEntrypoint logs are not silently discarded in CI - Fix misleading comment on upgradeCoordinatorLogger: the default is logr.Discard(), not stderr; SetUpgradeCoordinatorLogger enables real output - Switch Consistently from positional (upgradeDoneCtx, pollInterval) to .WithContext().WithPolling() to make the termination signal explicit and avoid the context being misread as a duration - Rename CheckinAndWait → CheckInAndWait for consistency with CheckIn/WaitForUpgrade; update upgrade_barrier.go, AGENTS.md, and region_upgrade_copy.go - Update main.go suite comment to describe the UpgradeCoordinator model instead of the now-removed runner-election model Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Patrik Suba (patriksuba) · 2026-07-21T07:03:00Z

/test images

… env vars Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

…ation

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

roi (roivaz) · 2026-07-21T15:30:57Z

/lgtm

openshift-ci · 2026-07-21T15:31:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, patriksuba, roivaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [deads2k,roivaz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-07-21T18:13:29Z

/retest-required

Remaining retests: 0 against base HEAD d276595 and 2 for PR HEAD a0aa33a in total

Andrii Klymenko (aklymenk) · 2026-07-21T22:06:14Z

/test e2e-parallel

openshift-merge-bot · 2026-07-22T00:16:13Z

/retest-required

Remaining retests: 0 against base HEAD a9ec591 and 1 for PR HEAD a0aa33a in total

openshift-merge-bot · 2026-07-22T02:58:16Z

/retest-required

Remaining retests: 0 against base HEAD b5c64b9 and 0 for PR HEAD a0aa33a in total

openshift-ci Bot added do-not-merge/work-in-progress approved labels Jun 19, 2026

Patrik Suba (patriksuba) mentioned this pull request Jun 19, 2026

ARO-HCP: add optional in-place upgrade E2E workflow openshift/release#80778

Merged

5 tasks

coderabbitai Bot mentioned this pull request Jun 24, 2026

ARO-HCP: add upgrade-path presubmit openshift/release#81044

Merged

3 tasks

Patrik Suba (patriksuba) changed the title ~~test: add upgrade/create and upgrade/post-infra E2E suites with node stability validation~~ test: add upgrade/in-place E2E suite with node pool stability validation Jun 26, 2026

Patrik Suba (patriksuba) marked this pull request as ready for review July 2, 2026 12:16

Copilot AI review requested due to automatic review settings July 2, 2026 12:16

openshift-ci Bot removed the do-not-merge/work-in-progress label Jul 2, 2026

openshift-ci Bot requested review from Ben Vesel (bennerv) and David Eads (deads2k) July 2, 2026 12:17

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 2, 2026 12:17 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

openshift-ci Bot removed the approved label Jul 2, 2026

Copilot AI review requested due to automatic review settings July 3, 2026 09:33

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 3, 2026 09:33 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Comment thread test/util/framework/make_runner.go Outdated

Comment thread test/cmd/aro-hcp-tests/main.go Outdated

Comment thread test/e2e/hypershiftoperator_upgrade.go Outdated

Copilot AI review requested due to automatic review settings July 9, 2026 15:00

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 9, 2026 15:00 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread test/e2e/hypershiftoperator_upgrade.go Outdated

Comment thread test/e2e/hypershiftoperator_upgrade.go Outdated

Comment thread test/util/framework/make_runner.go Outdated

Copilot AI review requested due to automatic review settings July 10, 2026 08:57

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 10, 2026 08:57 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread test/util/framework/make_runner.go Outdated

Comment thread test/util/framework/make_runner.go Outdated

Copilot AI review requested due to automatic review settings July 13, 2026 15:11

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 20, 2026 13:21 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 20, 2026 15:44

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 20, 2026 15:45 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

Comment thread test/e2e/region_upgrade.go Outdated

test: address haProxyImage container ambiguity and document local run…

e661cae

… env vars Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI review requested due to automatic review settings July 21, 2026 08:46

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 21, 2026 08:46 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread test/cmd/aro-hcp-tests/main.go

roi (roivaz) reviewed Jul 21, 2026

View reviewed changes

Comment thread test/e2e/region_upgrade.go

roi (roivaz) reviewed Jul 21, 2026

View reviewed changes

Comment thread test/util/framework/upgrade_coordinator.go Outdated

test: move helpers after Describe block and harden coordinator logger

cdbc989

Copilot AI review requested due to automatic review settings July 21, 2026 12:54

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 21, 2026 12:55 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread test/util/framework/upgrade_barrier.go

test: improve upgrade barrier diagnostics and document run-test limit…

a0aa33a

…ation

Copilot AI review requested due to automatic review settings July 21, 2026 13:49

Copilot started reviewing on behalf of Patrik Suba (patriksuba) July 21, 2026 13:49 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread test/util/framework/upgrade_coordinator.go

Comment thread test/util/framework/upgrade_barrier_test.go

openshift-ci Bot assigned roi (roivaz) Jul 21, 2026

openshift-ci Bot added the lgtm label Jul 21, 2026

openshift-merge-bot Bot merged commit 67e82f8 into Azure:main Jul 22, 2026
16 checks passed

Uh oh!

Conversation

Patrik Suba (patriksuba) commented Jun 19, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

New files

Modified files

Why

Testing

Special notes for your reviewer

PR Checklist

Uh oh!

openshift-ci Bot commented Jun 19, 2026

Uh oh!

Patrik Suba (patriksuba) commented Jun 19, 2026

Uh oh!

openshift-ci Bot commented Jun 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Patrik Suba (patriksuba) commented Jul 10, 2026

Uh oh!

Patrik Suba (patriksuba) commented Jul 10, 2026

Uh oh!

Patrik Suba (patriksuba) commented Jul 10, 2026

Uh oh!

Patrik Suba (patriksuba) commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Patrik Suba (patriksuba) commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Patrik Suba (patriksuba) commented Jun 19, 2026 •

edited by openshift-ci Bot

Loading