ARO-HCP: add optional in-place upgrade E2E workflow - #80778
ARO-HCP: add optional in-place upgrade E2E workflow#80778openshift-merge-bot[bot] merged 19 commits into
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
needs-rebase
|
@patriksuba, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
🚥 Pre-merge checks | ✅ 2 | ❌ 10❌ Failed checks (10 inconclusive)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
ci-operator/step-registry/aro-hcp/upgrade/infra/aro-hcp-upgrade-infra-ref.yaml (1)
20-22: Consider aligning ARO_HCP_DEPLOY_ENV defaults across upgrade chain steps.The upgrade-infra step defaults to
ci00while the post-upgrade step defaults toprow. Although the job-level env override (ARO_HCP_DEPLOY_ENV: ci01) prevents drift in the currente2e-parallel-hypershift-upgradejob, inconsistent step-level defaults could cause silent environment selection issues if the workflow is reused in other contexts without explicit env overrides.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/aro-hcp/upgrade/infra/aro-hcp-upgrade-infra-ref.yaml` around lines 20 - 22, The ARO_HCP_DEPLOY_ENV variable in the aro-hcp-upgrade-infra step defaults to "ci00" while the post-upgrade step defaults to "prow", creating inconsistency that could cause silent environment selection issues if the upgrade chain is reused without explicit environment overrides. Align the default value of ARO_HCP_DEPLOY_ENV in the upgrade-infra step to match the default used in the post-upgrade step to ensure consistent behavior across all upgrade chain steps regardless of job context.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/aro-hcp/git/checkout-head/aro-hcp-git-checkout-head-commands.sh`:
- Around line 13-14: The git fetch commands on lines 13-14 fail hard when the
ref is a commit SHA (from PULL_PULL_SHA in presubmits) because git cannot fetch
raw commit SHAs directly. Since the commit is already available locally after
clonerefs clones the PR, change both git fetch commands to soft-fail by
appending || true to each command, and add a verification step after the fetches
to check if the ref exists locally (using git rev-parse --verify "${ref}")
before attempting the git checkout to ensure the ref is available.
In
`@ci-operator/step-registry/aro-hcp/git/checkout/aro-hcp-git-checkout-commands.sh`:
- Around line 12-14: The git fetch command on line 12 fails hard when the remote
does not support SHA-based fetches, causing the entire script to exit due to set
-o errexit before the git checkout command can attempt to resolve a valid local
commit. Make both fetch attempts best-effort by ensuring they do not cause
script failure (using || true pattern), then add an explicit verification step
that validates the ref is locally resolvable before attempting the git checkout
on line 14. Apply the same pattern fix to aro-hcp-git-checkout-head-commands.sh
which has the identical issue.
In
`@ci-operator/step-registry/aro-hcp/test/local-post-upgrade/aro-hcp-test-local-post-upgrade-ref.yaml`:
- Around line 30-32: The ARO_HCP_DEPLOY_ENV parameter in the post-upgrade step
defaults to "prow" but should be aligned with the pre-upgrade step which
defaults to "ci00". Change the default value of the ARO_HCP_DEPLOY_ENV variable
from "prow" to "ci00" to ensure consistency between the pre-upgrade and
post-upgrade workflow steps and prevent configuration mismatches when this step
is reused in other workflows.
In
`@ci-operator/step-registry/aro-hcp/test/local-pre-upgrade/aro-hcp-test-local-pre-upgrade-commands.sh`:
- Around line 53-54: The fallback value for the junit_shared_name variable on
line 53 uses the same default name as the post-upgrade step's fallback, which
causes artifact collision when E2E_JUNIT_SHARED_NAME is not explicitly set.
Change the default fallback from "junit-e2e-suite.xml.gz" to a
pre-upgrade-specific name (for example, by adding "pre-upgrade" to the filename)
so that the pre-upgrade and post-upgrade steps each write to distinct artifact
files and prevent the post-upgrade from overwriting the pre-upgrade output.
---
Nitpick comments:
In
`@ci-operator/step-registry/aro-hcp/upgrade/infra/aro-hcp-upgrade-infra-ref.yaml`:
- Around line 20-22: The ARO_HCP_DEPLOY_ENV variable in the
aro-hcp-upgrade-infra step defaults to "ci00" while the post-upgrade step
defaults to "prow", creating inconsistency that could cause silent environment
selection issues if the upgrade chain is reused without explicit environment
overrides. Align the default value of ARO_HCP_DEPLOY_ENV in the upgrade-infra
step to match the default used in the post-upgrade step to ensure consistent
behavior across all upgrade chain steps regardless of job context.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 355acf0a-a1bf-4e36-a534-176f3d584248
⛔ Files ignored due to path filters (1)
ci-operator/jobs/Azure/ARO-HCP/Azure-ARO-HCP-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (26)
ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yamlci-operator/step-registry/aro-hcp/git/checkout-head/OWNERSci-operator/step-registry/aro-hcp/git/checkout-head/aro-hcp-git-checkout-head-commands.shci-operator/step-registry/aro-hcp/git/checkout-head/aro-hcp-git-checkout-head-ref.metadata.jsonci-operator/step-registry/aro-hcp/git/checkout-head/aro-hcp-git-checkout-head-ref.yamlci-operator/step-registry/aro-hcp/git/checkout/OWNERSci-operator/step-registry/aro-hcp/git/checkout/aro-hcp-git-checkout-commands.shci-operator/step-registry/aro-hcp/git/checkout/aro-hcp-git-checkout-ref.metadata.jsonci-operator/step-registry/aro-hcp/git/checkout/aro-hcp-git-checkout-ref.yamlci-operator/step-registry/aro-hcp/local-e2e-upgrade/OWNERSci-operator/step-registry/aro-hcp/local-e2e-upgrade/aro-hcp-local-e2e-upgrade-workflow.metadata.jsonci-operator/step-registry/aro-hcp/local-e2e-upgrade/aro-hcp-local-e2e-upgrade-workflow.yamlci-operator/step-registry/aro-hcp/test/local-post-upgrade/OWNERSci-operator/step-registry/aro-hcp/test/local-post-upgrade/aro-hcp-test-local-post-upgrade-commands.shci-operator/step-registry/aro-hcp/test/local-post-upgrade/aro-hcp-test-local-post-upgrade-ref.metadata.jsonci-operator/step-registry/aro-hcp/test/local-post-upgrade/aro-hcp-test-local-post-upgrade-ref.yamlci-operator/step-registry/aro-hcp/test/local-pre-upgrade/OWNERSci-operator/step-registry/aro-hcp/test/local-pre-upgrade/aro-hcp-test-local-pre-upgrade-commands.shci-operator/step-registry/aro-hcp/test/local-pre-upgrade/aro-hcp-test-local-pre-upgrade-ref.metadata.jsonci-operator/step-registry/aro-hcp/test/local-pre-upgrade/aro-hcp-test-local-pre-upgrade-ref.yamlci-operator/step-registry/aro-hcp/upgrade/infra/OWNERSci-operator/step-registry/aro-hcp/upgrade/infra/aro-hcp-upgrade-infra-commands.shci-operator/step-registry/aro-hcp/upgrade/infra/aro-hcp-upgrade-infra-ref.metadata.jsonci-operator/step-registry/aro-hcp/upgrade/infra/aro-hcp-upgrade-infra-ref.yamlcore-services/prow/02_config/_boskos.yamlcore-services/prow/02_config/generate-boskos.py
3b0a609 to
86d532f
Compare
openshift-ci
Bot
added
needs-rebase
openshift-ci
Bot
removed
the
needs-rebase
|
@patriksuba, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci-operator/step-registry/aro-hcp/test/local-upgrade/aro-hcp-test-local-upgrade-commands.sh (1)
44-55: 🎯 Functional Correctness | 🔴 CriticalRemove sensitive data logging on line 53.
The
cat "${OVERRIDE_CONFIG_FILE}"command on line 53 logs the entire override configuration file contents to CI logs. Per the security best practices for step registry scripts, avoid printing passwords, tokens, cluster URLs, or kubeconfig contents. The override file likely contains sensitive hypershift configuration, subscription IDs, and cluster endpoints.Replace the
catcommand with a summary log:echo "Created upgrade override at: ${OVERRIDE_CONFIG_FILE}"Remove the line logging file contents or limit it to file size/metadata if debugging is needed, ensuring sensitive data is never exposed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/aro-hcp/test/local-upgrade/aro-hcp-test-local-upgrade-commands.sh` around lines 44 - 55, The upgrade helper script is printing the full override file contents via the `cat "${OVERRIDE_CONFIG_FILE}"` command, which can expose sensitive hypershift/config data in CI logs. Remove that content dump from `aro-hcp-test-local-upgrade-commands.sh` and keep only the existing summary message around `OVERRIDE_CONFIG_FILE`; if debugging is needed, log only non-sensitive metadata and avoid printing file contents.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In
`@ci-operator/step-registry/aro-hcp/test/local-upgrade/aro-hcp-test-local-upgrade-commands.sh`:
- Around line 44-55: The upgrade helper script is printing the full override
file contents via the `cat "${OVERRIDE_CONFIG_FILE}"` command, which can expose
sensitive hypershift/config data in CI logs. Remove that content dump from
`aro-hcp-test-local-upgrade-commands.sh` and keep only the existing summary
message around `OVERRIDE_CONFIG_FILE`; if debugging is needed, log only
non-sensitive metadata and avoid printing file contents.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 2a31c93b-5048-4f72-8686-f6338e323201
📒 Files selected for processing (7)
ci-operator/step-registry/aro-hcp/git/checkout-head/aro-hcp-git-checkout-head-commands.shci-operator/step-registry/aro-hcp/git/checkout/aro-hcp-git-checkout-commands.shci-operator/step-registry/aro-hcp/local-e2e-upgrade/aro-hcp-local-e2e-upgrade-workflow.yamlci-operator/step-registry/aro-hcp/test/local-upgrade/OWNERSci-operator/step-registry/aro-hcp/test/local-upgrade/aro-hcp-test-local-upgrade-commands.shci-operator/step-registry/aro-hcp/test/local-upgrade/aro-hcp-test-local-upgrade-ref.metadata.jsonci-operator/step-registry/aro-hcp/test/local-upgrade/aro-hcp-test-local-upgrade-ref.yaml
💤 Files with no reviewable changes (1)
- ci-operator/step-registry/aro-hcp/test/local-upgrade/OWNERS
🚧 Files skipped from review as they are similar to previous changes (2)
- ci-operator/step-registry/aro-hcp/git/checkout-head/aro-hcp-git-checkout-head-commands.sh
- ci-operator/step-registry/aro-hcp/git/checkout/aro-hcp-git-checkout-commands.sh
|
@patriksuba, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
d6d5bc4 to
1d03ed5
Compare
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-hypershift-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse ack |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
rehearsals-ack
|
/test ci-operator-config-metadata |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-hypershift-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-hypershift-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@patriksuba: |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-inplace-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-hypershift-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@patriksuba: job(s): pull-ci-Azure-ARO-HCP-main-e2e-parallel-hypershift-upgrade either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-inplace-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@patriksuba: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-e2e-parallel-inplace-upgrade |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse ack |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
14ecf0b to
8fe91a8
Compare
openshift-merge-bot
Bot
removed
the
rehearsals-ack
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse ack |
|
@patriksuba: your |
|
/pj-rehearse ack |
|
@patriksuba: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: patriksuba, roivaz The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
e9cb643
into
openshift:main
3 participants
ARO-27647
What
Adds an optional, manual-only presubmit and step-registry workflow for in-place regional infrastructure upgrade E2E on ARO-HCP (svc + mgmt components, including Hypershift — not Hypershift-only).
Job:
e2e-parallel-inplace-upgrade→ workflowaro-hcp-local-e2e-upgradeoptional: truerun_if_changed: ^$(manual/test e2e-parallel-inplace-upgradeonly)ARO HCP E2E Hosted Clusters (EA Subscription),ARO HCP E2E Hosted Clusters 2 (EA Subscription),ARO HCP E2E Hosted Clusters - Dev - 02,ARO HCP E2E Hosted Clusters - Dev - 03/westus3/ci01Workflow phases
aro-hcp-provision-from-mainchecks out main, resolves ACR images fromimages-push(commit SHA tags), provisions regional infraaro-hcp-test-local-upgrade(PR head via clonerefs) buildsOVERRIDE_CONFIG_FILEfrom PR pipeline images + Hypershift digests from PRconfig/config.yaml, exportsUPGRADE_SPEC_COUNT, runs hardcodedupgrade/in-placearo-hcp-test-localruns the regular local suite against the upgraded environmentStep-registry pieces
aro-hcp-local-e2e-upgrade— workflow abovearo-hcp-test-local-upgrade— PR image override +upgrade/in-placesuite (suite drivespipeline/RP.*; CI does not re-fetch kubeconfigs for make targets beyond harness setup)aro-hcp-provision-from-main(shared withupgrade-e2e)provision-completeorprovision-from-main-completeRemoved earlier draft approach: dedicated
aro-hcp-git-checkout*steps andPROVISION_GIT_REFcheckout/skip-override onwrite-config/provision-environment. Baseline is now the shared from-main path; PR head is the default clone for the test steps.Why
We need CI that validates in-place upgrade of a main baseline to PR images, then confirms the upgraded stack still passes regular local e2e — not only greenfield provision on a single ref.
upgrade/in-placekeeps create → upgrade → validation (e.g. node pool / rollout expectations) in one suite so cluster state stays in-processtest-localafterward catches regressions that only show up after upgradeDistinct from
upgrade-e2e-parallel/aro-hcp-upgrade-e2e, which re-entersentrypoint/Regionwith PR images (ARO_HCP_PROVISION_ABORT_IF_EXISTS=false). This job upgrades via the e2e suite’s pipeline targets on an already-provisioned region.Companion work
Job stays optional until companion work in Azure/ARO-HCP is solid (e.g.
upgrade/in-placesuite). Dedicated upgrade Boskos slot / subscription leasing remains deferred.Test plan
/test e2e-parallel-inplace-upgradeon an ARO-HCP PR that exercises the suiteprovision-from-main(main + ACR), not PR pipeline digestsupgrade/in-placetest-localruns after upgrade and both junit artifacts appear under their stepsprovision-from-main-completeis present