ARO-HCP: add upgrade-path presubmit - #81044
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a from-main provisioning step, a new upgrade-e2e workflow and optional presubmit job, and a configurable flag to skip aborting when regional infrastructure already exists. ChangesARO-HCP upgrade flow
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CIConfig as Azure-ARO-HCP-main.yaml
participant Workflow as aro-hcp-upgrade-e2e
participant Baseline as aro-hcp-provision-from-main
participant Upgrade as aro-hcp-upgrade-environment
participant Tests as aro-hcp-test-local
CIConfig->>Workflow: run upgrade-e2e-parallel
Workflow->>Baseline: provision baseline from main
Workflow->>Upgrade: provision PR upgrade environment
Workflow->>Tests: run local e2e tests
Suggested labels: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
gmfrasca
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci-operator/config/Azure/ARO-HCP/README.md`:
- Line 11: The README’s ARO-HCP documentation link is broken because it points
to a non-existent path in Azure/ARO-HCP@main. Update the link in the README to a
real existing document, or add the referenced doc as part of this change, and
make sure the `upgrade-e2e-parallel` description still points users to the
correct usage and failure-interpretation guidance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 4d8539a6-3309-4a5f-934d-f87e2737b5ec
⛔ Files ignored due to path filters (1)
ci-operator/jobs/Azure/ARO-HCP/Azure-ARO-HCP-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (11)
ci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yamlci-operator/config/Azure/ARO-HCP/README.mdci-operator/step-registry/aro-hcp/provision/environment/aro-hcp-provision-environment-commands.shci-operator/step-registry/aro-hcp/provision/environment/aro-hcp-provision-environment-ref.yamlci-operator/step-registry/aro-hcp/provision/from-main/OWNERSci-operator/step-registry/aro-hcp/provision/from-main/aro-hcp-provision-from-main-commands.shci-operator/step-registry/aro-hcp/provision/from-main/aro-hcp-provision-from-main-ref.metadata.jsonci-operator/step-registry/aro-hcp/provision/from-main/aro-hcp-provision-from-main-ref.yamlci-operator/step-registry/aro-hcp/upgrade-e2e/OWNERSci-operator/step-registry/aro-hcp/upgrade-e2e/aro-hcp-upgrade-e2e-workflow.metadata.jsonci-operator/step-registry/aro-hcp/upgrade-e2e/aro-hcp-upgrade-e2e-workflow.yaml
| #!/bin/bash | ||
| set -o errexit | ||
| set -o nounset | ||
| set -o pipefail | ||
|
|
||
| env_file="${SHARED_DIR}/aro-hcp-slot.env" | ||
| if [[ -f "${env_file}" ]]; then | ||
| # shellcheck disable=SC1090 | ||
| source "${env_file}" | ||
| fi | ||
|
|
||
| export LOCATION="${SELECTED_LOCATION:-${LOCATION:-}}" | ||
| : "${LOCATION:?LOCATION must be provided by SELECTED_LOCATION or the legacy runtime slot export file}" | ||
|
|
||
| export CLUSTER_PROFILE_DIR="/var/run/aro-hcp-${VAULT_SECRET_PROFILE}" | ||
|
|
||
| export AZURE_CLIENT_ID; AZURE_CLIENT_ID=$(cat "${CLUSTER_PROFILE_DIR}/client-id") | ||
| export AZURE_TENANT_ID; AZURE_TENANT_ID=$(cat "${CLUSTER_PROFILE_DIR}/tenant") | ||
| export AZURE_CLIENT_SECRET; AZURE_CLIENT_SECRET=$(cat "${CLUSTER_PROFILE_DIR}/client-secret") | ||
| INFRA_SUBSCRIPTION_ID=$(cat "${CLUSTER_PROFILE_DIR}/infra-${ARO_HCP_DEPLOY_ENV}-subscription-id") | ||
| export INFRA_SUBSCRIPTION_ID | ||
| export DEPLOY_ENV="${ARO_HCP_DEPLOY_ENV}" | ||
| export AZURE_TOKEN_CREDENTIALS=prod | ||
|
|
||
| az login --service-principal -u "${AZURE_CLIENT_ID}" -p "${AZURE_CLIENT_SECRET}" --tenant "${AZURE_TENANT_ID}" --output none | ||
| az account set --subscription "${INFRA_SUBSCRIPTION_ID}" | ||
| oc version | ||
| kubelogin --version | ||
|
|
||
| # Check out main branch to provision the baseline environment. | ||
| # The container image has the PR source baked in; we swap to main so that | ||
| # Bicep templates, Helm charts, config, and pipeline definitions all come | ||
| # from the current state of the default branch. | ||
| echo "Fetching and checking out origin/main for baseline provision ..." | ||
| git fetch origin main | ||
| git checkout FETCH_HEAD | ||
| echo "Checked out main at $(git rev-parse --short HEAD)" | ||
|
|
||
| OVERRIDE_CONFIG_FILE="${SHARED_DIR}/config-override.yaml" | ||
|
|
||
| # MSI mock SP overrides (if provided) — needed for both baseline and upgrade | ||
| if [[ -n "${LEASED_MSI_MOCK_SP:-}" ]]; then | ||
| MSI_MOCK_CLIENT_ID=$(yq ".miMockPool.\"${LEASED_MSI_MOCK_SP}\".clientId" dev-infrastructure/openshift-ci/msi-mock-pool.yaml) | ||
| MSI_MOCK_PRINCIPAL_ID=$(yq ".miMockPool.\"${LEASED_MSI_MOCK_SP}\".principalId" dev-infrastructure/openshift-ci/msi-mock-pool.yaml) | ||
| MSI_MOCK_CERT_NAME=$(yq ".miMockPool.\"${LEASED_MSI_MOCK_SP}\".certName" dev-infrastructure/openshift-ci/msi-mock-pool.yaml) | ||
| if [[ -z "${MSI_MOCK_CLIENT_ID}" || "${MSI_MOCK_CLIENT_ID}" == "null" || \ | ||
| -z "${MSI_MOCK_PRINCIPAL_ID}" || "${MSI_MOCK_PRINCIPAL_ID}" == "null" || \ | ||
| -z "${MSI_MOCK_CERT_NAME}" || "${MSI_MOCK_CERT_NAME}" == "null" ]]; then | ||
| echo "ERROR: LEASED_MSI_MOCK_SP='${LEASED_MSI_MOCK_SP}' not found in dev-infrastructure/openshift-ci/msi-mock-pool.yaml" | ||
| exit 1 | ||
| fi | ||
| echo "MSI mock SP override: ${LEASED_MSI_MOCK_SP} -> clientId=${MSI_MOCK_CLIENT_ID}" | ||
| yq eval -n " | ||
| .clouds.dev.environments.${DEPLOY_ENV}.defaults.miMockClientId = \"${MSI_MOCK_CLIENT_ID}\" | | ||
| .clouds.dev.environments.${DEPLOY_ENV}.defaults.miMockPrincipalId = \"${MSI_MOCK_PRINCIPAL_ID}\" | | ||
| .clouds.dev.environments.${DEPLOY_ENV}.defaults.miMockCertName = \"${MSI_MOCK_CERT_NAME}\" | ||
| " > "${OVERRIDE_CONFIG_FILE}" | ||
| else | ||
| echo "No MSI mock SP lease provided, skipping mock SP overrides" | ||
| echo "{}" > "${OVERRIDE_CONFIG_FILE}" | ||
| fi | ||
|
|
||
| # Temporary MGMT cluster sizing overrides for single-wave E2E parallelism. | ||
| # These will be removed once the matching config.yaml defaults land in ARO-HCP. | ||
| # Only apply when identity containers are leased (E2E runs); healthcheck | ||
| # workflows provision without leases and should use the default sizing. | ||
| if [[ -n "${LEASED_MSI_CONTAINERS:-}" ]]; then | ||
| yq -i " | ||
| .clouds.dev.environments.${DEPLOY_ENV}.defaults.mgmt.aks.userAgentPool.minCount = 7 | | ||
| .clouds.dev.environments.${DEPLOY_ENV}.defaults.mgmt.aks.infraAgentPool.vmSize = \"Standard_D8ds_v6\" | ||
| " "${OVERRIDE_CONFIG_FILE}" | ||
| else | ||
| yq -i " | ||
| .clouds.dev.environments.${DEPLOY_ENV}.defaults.mgmt.aks.userAgentPool.minCount = 1 | ||
| " "${OVERRIDE_CONFIG_FILE}" | ||
| fi | ||
|
|
||
| echo "Created override config at: ${OVERRIDE_CONFIG_FILE}" | ||
| cat "${OVERRIDE_CONFIG_FILE}" | ||
|
|
||
| CONFIG_PROV="${SHARED_DIR}/config-prov.yaml" | ||
|
|
||
| finalize() { | ||
| if [[ -s "${CONFIG_PROV}" ]]; then | ||
| mv "${CONFIG_PROV}" "${SHARED_DIR}/config.yaml" | ||
| cp "${SHARED_DIR}/config.yaml" "${ARTIFACT_DIR}/config.yaml" | ||
| fi | ||
| } | ||
| trap finalize EXIT | ||
|
|
||
| unset GOFLAGS | ||
| make -o tooling/templatize/templatize entrypoint/Region \ | ||
| DEPLOY_ENV="${DEPLOY_ENV}" \ | ||
| OVERRIDE_CONFIG_FILE="${OVERRIDE_CONFIG_FILE}" \ | ||
| EXTRA_ARGS="--region ${LOCATION} --abort-if-regional-exist" \ | ||
| TIMING_OUTPUT=${SHARED_DIR}/steps.yaml.gz \ | ||
| ENTRYPOINT_JUNIT_OUTPUT=${ARTIFACT_DIR}/junit_entrypoint_baseline.xml \ | ||
| CONFIG_OUTPUT=${CONFIG_PROV} | ||
|
|
||
| touch "${SHARED_DIR}/provision-from-main-complete" | ||
| echo "Baseline provision from main complete." |
There was a problem hiding this comment.
Could we tweak and reuse the same provision script everywhere? This is a lot of duplication.
There was a problem hiding this comment.
So ideally, to reuse the provision script we'd want to centralize it in a library script, but step-registry only injects *-commands.sh into the pod - other files in the registry directory aren't mounted at runtime so the library wouldn't be available.
With that said, the ARO-HCP repo itself is also mounted, so we can put a shared provision script there (e.g. dev-infrastructure/openshift-ci/lib/provision-common.sh) and have each *-commands.sh source it. I think that would be better suited for a follow-up task though, as we should have provision-global-pipeline, provision-environment, and provision-from-main all switch over to the centralized provision script in the same PR
There was a problem hiding this comment.
discussed in-person: Updated the (currently un-utilized) provision script[1] that was already in ARO-HCP with recent changes. Once that's in, we can leverage it here and in the provision-environment step, which I propose doing in a separate PR:
There was a problem hiding this comment.
Do you want to merge the script as-is now or wait for Azure/ARO-HCP#5866 to land? Either option is fine with me.
There was a problem hiding this comment.
i'd say lets merge this as-is - Azure/ARO-HCP#5866 touches a lot more than just this optional job, so let's get some real-world usage out of it as a sort-of validation step before centralizing, since that would have a bigger blast radius
59eb264 to
91696d9
Compare
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
91696d9 to
f03f8f8
Compare
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
f03f8f8 to
18eb260
Compare
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
18eb260 to
c0ba249
Compare
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/retest |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel |
|
/retest |
|
@gmfrasca: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
| workflow: aro-hcp-local-e2e | ||
| - as: upgrade-e2e-parallel | ||
| optional: true | ||
| run_if_changed: ^NEVER_AUTO_TRIGGER$ |
There was a problem hiding this comment.
You can achieve the same on-demand only behavior with always_run: false and drop the run_if_changed.
There was a problem hiding this comment.
ah interesting - the original config used optional: true without run_if_changed at all, which was apparently an unsupported configuration and failed a job config ci check for not having run_if_changed. If always_run: false satisfies that requirement that would be much more preferable than the unmatchable regex hack used here. Will update see if the config check passes
| - **`Azure-ARO-HCP-main__periodic.yaml`** - Scheduled periodic jobs (cluster creation, E2E tests, cleanup) | ||
| - **`Azure-ARO-HCP-main__image-updater.yaml`** - Automated image digest update job - Mon-Fri | ||
|
|
||
| The `upgrade-e2e-parallel` optional presubmit validates main-to-PR infrastructure upgrades. See [ARO-HCP docs](https://github.com/Azure/ARO-HCP/blob/main/docs/ci-upgrade-path-presubmit.md) for usage and failure interpretation. |
There was a problem hiding this comment.
This doc is completely out of date. All our CI docs live in the ARO-HCP repo, we should just remove this whole doc.
There was a problem hiding this comment.
noted. for a cleaner git history, i think i should remove just this block (so no net changes made here), and i'll open another PR to simply remove this entire doc. Will also move the full doc referenced here (added by Azure/ARO-HCP#5780) to the ci/ subdir
716141d to
88565d5
Compare
Add an optional presubmit that validates main-to-PR infrastructure upgrades by provisioning from main then upgrading to the PR branch. - New step: aro-hcp-provision-from-main (baseline provision from main) - New workflow: aro-hcp-upgrade-e2e (provision → upgrade → e2e) - Parameterize --abort-if-regional-exist in provision-environment Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
…images Replace pipeline image dependency injection with direct ACR lookup by main's commit SHA, matching the tag format used by images-push postsubmit. Includes retry loop with parent-commit fallback for race conditions. Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
88565d5 to
969cea7
Compare
| # The CSPR postsubmit pushes service images to ACR tagged with the 7-char | ||
| # commit SHA (see aro-hcp-images-push step). Resolve the tag and look up | ||
| # the digest for each service image so the baseline uses main's actual images. |
There was a problem hiding this comment.
This comment is inaccurate and should be fixed to avoid misleading future maintainers.
Images are not pushed to ACR by the CSPR postsubmit. They're pushed by the generic images-push-postsubmit job, which runs the aro-hcp-images-push step with no env override, so DEPLOY_ENV = the ref default dev. That step mirrors the CI-built service images into the shared arohcpsvcdev ACR tagged with the 7-char commit SHA.
cspr-pipeline-postsubmit is a separate job that deploys the Clusters Service PR-check sandbox environment (DEPLOY_ENV=cspr); it doesn't own the image push, and per docs/cspr.md it uses the same shared arohcpsvcdev ACR anyway.
The resolution here still works only because every dev env (including ci01) shares that one arohcpsvcdev ACR and the same repos, so the ci01-vs-dev DEPLOY_ENV divergence is masked. Suggest:
- Reword the comment to reference the generic
images-push-postsubmitjob /aro-hcp-images-pushstep (pushes toarohcpsvcdevby 7-char SHA), and drop the CSPR reference. - Optionally resolve the ACR/repo coordinates with
DEPLOY_ENV=dev(matchingaro-hcp-images-push) instead of reusing the provisioningARO_HCP_DEPLOY_ENV=ci01, so the dependency is explicit rather than relying on the shared-ACR coincidence.
There was a problem hiding this comment.
Done. Reworded the comment to reference images-push-postsubmit / aro-hcp-images-push, and switched ACR/repo resolution to read from the dev config (IMAGES_DEPLOY_ENV="dev") since that's what images-push actually uses instead of implicitly relying on ci01 happening to share the same ACR.
…walk-back Replace single-commit polling + parent fallback with a two-phase approach: poll HEAD for 15m, then walk back up to 20 commits to find the newest main SHA with images already pushed to ACR. Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
969cea7 to
ea0b15c
Compare
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@gmfrasca: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gmfrasca, roivaz The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/pj-rehearse ack |
|
@roivaz: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
Replace the "latest by timestamp" image resolution strategy with SHA-anchored resolution matching the pattern from PR openshift#81044. The images-push-postsubmit job tags service images with the 7-char commit SHA on every merge to ARO-HCP main. Resolving by SHA guarantees version coherence across all services and avoids picking up broken/in-progress images from dev ACR. Includes polling (up to 15min) for the HEAD commit's images and walk-back through recent history if HEAD's images aren't available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(step-registry): add upgrade-path presubmit for ARO HCP (ARO-27255) Add an optional presubmit that validates main-to-PR infrastructure upgrades by provisioning from main then upgrading to the PR branch. - New step: aro-hcp-provision-from-main (baseline provision from main) - New workflow: aro-hcp-upgrade-e2e (provision → upgrade → e2e) - Parameterize --abort-if-regional-exist in provision-environment Signed-off-by: Giulio Frasca <gfrasca@redhat.com> * ARO-HCP upgrade-from-main:use images from ACR instead of CI-built PR images Replace pipeline image dependency injection with direct ACR lookup by main's commit SHA, matching the tag format used by images-push postsubmit. Includes retry loop with parent-commit fallback for race conditions. Signed-off-by: Giulio Frasca <gfrasca@redhat.com> * ARO-HCP: from-main upgrade resolve main images from ACR with history walk-back Replace single-commit polling + parent fallback with a two-phase approach: poll HEAD for 15m, then walk back up to 20 commits to find the newest main SHA with images already pushed to ACR. Signed-off-by: Giulio Frasca <gfrasca@redhat.com> --------- Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
* feat(step-registry): add upgrade-path presubmit for ARO HCP (ARO-27255) Add an optional presubmit that validates main-to-PR infrastructure upgrades by provisioning from main then upgrading to the PR branch. - New step: aro-hcp-provision-from-main (baseline provision from main) - New workflow: aro-hcp-upgrade-e2e (provision → upgrade → e2e) - Parameterize --abort-if-regional-exist in provision-environment Signed-off-by: Giulio Frasca <gfrasca@redhat.com> * ARO-HCP upgrade-from-main:use images from ACR instead of CI-built PR images Replace pipeline image dependency injection with direct ACR lookup by main's commit SHA, matching the tag format used by images-push postsubmit. Includes retry loop with parent-commit fallback for race conditions. Signed-off-by: Giulio Frasca <gfrasca@redhat.com> * ARO-HCP: from-main upgrade resolve main images from ACR with history walk-back Replace single-commit polling + parent fallback with a two-phase approach: poll HEAD for 15m, then walk back up to 20 commits to find the newest main SHA with images already pushed to ACR. Signed-off-by: Giulio Frasca <gfrasca@redhat.com> --------- Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
![openshift-prow[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
* feat(step-registry): add upgrade-path presubmit for ARO HCP (ARO-27255) Add an optional presubmit that validates main-to-PR infrastructure upgrades by provisioning from main then upgrading to the PR branch. - New step: aro-hcp-provision-from-main (baseline provision from main) - New workflow: aro-hcp-upgrade-e2e (provision → upgrade → e2e) - Parameterize --abort-if-regional-exist in provision-environment Signed-off-by: Giulio Frasca <gfrasca@redhat.com> * ARO-HCP upgrade-from-main:use images from ACR instead of CI-built PR images Replace pipeline image dependency injection with direct ACR lookup by main's commit SHA, matching the tag format used by images-push postsubmit. Includes retry loop with parent-commit fallback for race conditions. Signed-off-by: Giulio Frasca <gfrasca@redhat.com> * ARO-HCP: from-main upgrade resolve main images from ACR with history walk-back Replace single-commit polling + parent fallback with a two-phase approach: poll HEAD for 15m, then walk back up to 20 commits to find the newest main SHA with images already pushed to ACR. Signed-off-by: Giulio Frasca <gfrasca@redhat.com> --------- Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
2 participants
Summary
Adds an optional presubmit job (
upgrade-e2e-parallel) that validates infrastructure upgrades frommainto the PR branch, catching regressions that only appear when existing environments are upgraded rather than freshly provisioned.aro-hcp-provision-from-main— checks outorigin/mainand provisions a baseline environment using main's Bicep/Helm/configaro-hcp-upgrade-e2e— orchestrates baseline provision → upgrade → e2e test → deprovisionaro-hcp-provision-environment— parameterized--abort-if-regional-existviaARO_HCP_PROVISION_ABORT_IF_EXISTSso the upgrade phase can re-deploy over the baselineupgrade-e2e-parallelinAzure-ARO-HCP-main.yaml— optional, auto-triggers ondev-infrastructure/,config/,*pipeline.yaml,*.bicep,*.bicepparamchangesHow it works
provision-from-mainfetches and checks outmain, provisions the environment with main's templates (no CI image overrides)aro-hcp-upgrade-environment(aliasedprovision-environment) runs in a fresh container with PR source, applies CI-built image overrides, and re-runsmake entrypoint/Region— ARM idempotency means only changed resources are updatedaro-hcp-test-localruns the full e2e suite against the upgraded environmentTesting
/test upgrade-e2e-parallelon this PR or an ARO-HCP PRprovision-from-mainfromaro-hcp-upgrade-environmente2e-parallelandupgrade-e2e-parallelcoexist without Boskos contentionJIRA: ARO-27255
Summary by CodeRabbit
This PR enhances the OpenShift CI/Azure ARO-HCP infrastructure pipelines by adding an optional presubmit job,
upgrade-e2e-parallel, to validate the full infrastructure upgrade flow frommainto the PR branch.Key changes:
upgrade-e2e-parallel) inci-operator/config/Azure/ARO-HCP/Azure-ARO-HCP-main.yamlaro-hcp-upgrade-e2eworkflow.dev-infrastructure/,config/,*pipeline.yaml,*.bicep, and*.bicepparam.run_if_changed: ^NEVER_AUTO_TRIGGER$).ARO_HCP_CLOUD=dev,ARO_HCP_DEPLOY_ENV=ci01ARO_HCP_PROVISION_ABORT_IF_EXISTS="false"(so the upgrade phase can redeploy over the baseline)ARO_HCP_SUITE_PARALLELISM="55"MULTISTAGE_PARAM_OVERRIDE_LOCATION=westus3aro-hcp-provision-from-mainprovisions a baseline by checking outorigin/mainwhile using PR-built image digests from CI inputs, generatingconfig-override.yaml(including optional MSI mock SP overrides and optional MGMT AKS sizing overrides), templating viamake ... templatize, and emitting baseline artifacts/config for the next phase.aro-hcp-upgrade-e2eworkflow:LEASED_MSI_MOCK_SPlease.aro-hcp-provision-from-main).aro-hcp-upgrade-environment→aro-hcp-provision-environment.aro-hcp-test-local).aro-hcp-provision-environmentto introduceARO_HCP_PROVISION_ABORT_IF_EXISTS(default"true").--abort-if-regional-existonly whenARO_HCP_PROVISION_ABORT_IF_EXISTSis"true", enabling redeploy-on-top behavior for upgrade workflows.from-mainandupgrade-e2esteps/workflow.ci-operator/config/Azure/ARO-HCP/README.md./pj-rehearse pull-ci-Azure-ARO-HCP-main-upgrade-e2e-parallel