[CI] Add parity auto-trigger workflow#3231
Conversation
|
Jenkins build for f4dfbd8845f2d05dd28225ca78af48c1926d9e31 commit finished as FAILURE |
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow to automatically scan recent completed trunk.yml push runs on main, determine when the relevant ROCm + CUDA check-runs for a given upstream SHA are fully complete, and then dispatch parity.yml (with a PR-only dry-run mode).
Changes:
- Introduces a scheduled (every 10 minutes) + manual + PR dry-run “parity auto-trigger” workflow.
- Implements SHA deduplication by checking existing
parity.ymlrun titles in the current repo. - Gates dispatch on completion of ROCm arch shard check-runs (only for arch workflows detected as having run) plus specific CUDA check-runs used by parity.
| # Pull recent parity runs. Run titles look like: | ||
| # "<csv_name or SHA> · mi355, mi300, mi200" | ||
| # Once any parity run exists for a SHA, we do not dispatch another | ||
| # report for that SHA. This keeps the dashboard to one report per | ||
| # upstream commit. | ||
| EXISTING=$(gh run list \ | ||
| --repo "$GITHUB_REPOSITORY" \ | ||
| --workflow parity.yml \ | ||
| --limit 1000 \ | ||
| --json displayTitle 2>/dev/null || echo '[]') | ||
|
|
||
| sha_already_dispatched() { | ||
| local sha="$1" | ||
| echo "$EXISTING" | jq -e --arg sha "$sha" \ | ||
| 'any(.[]; .displayTitle | contains($sha))' >/dev/null |
There was a problem hiding this comment.
Since the same parity.yml can be triggered outside of the auto-parity, we should narrow this down to consider only the auto-parity-triggered runs of parity.yml. In addition to SHA, one more "variable" is the arch list. For simplicity, let us get rid of the arch list workflow input entirely in auto-parity, so that we don't have to worry about whether a particular run covered all the archs we wanted.
There was a problem hiding this comment.
That being said, I do prefer the suggested version by Copilot review above, since it gets rid of the magic number of 1000 and seems to make the window logic more robust by using the right API parameters.
There was a problem hiding this comment.
Reworked. The dedup set is no longer gh run list --limit 1000; it is a REST query of parity.yml workflow_dispatch runs via --paginate with a created>=<max_age window> filter, so the dedup window always covers the scan window (no fixed 1000 cap).
There was a problem hiding this comment.
Dedup now considers only auto-parity-created runs: matches require the autoparity- run-name prefix or the github-actions[bot] actor, so manual parity.yml runs no longer suppress automation. For the arch concern, this PR is scoped to trunk/mi355 (archs defaults to mi355) and the dedup key is SHA + auto-parity signal rather than the arch list, so partial-arch coverage cannot cause a missed/duplicate dispatch. The archs input is retained (defaulted) but no longer factors into dedup; happy to drop it entirely if you prefer for the trunk-only scope.
There was a problem hiding this comment.
Done — implemented the Copilot suggestion: REST API with --paginate + a created>= time filter, dropping the magic 1000. The dedup window now derives from max_age_hours.
jithunnair-amd
left a comment
jithunnair-amd
left a comment
There was a problem hiding this comment.
@ethanwee1 I may not be able to help get this merged once you address the review comments, so please take Pruthvi's help for that. However, please do fix some of the issues pointed out in the review comments and make the inputs interface a bit simpler for the average user (please work with @pablo-garay to iterate on this).
I'm also okay with having the cron-scheduled runs only covering trunk in this PR, since that's our top requirement, and handle the complexity of other archs/workflows in a follow-up PR.
Thanks!
| # Pull recent parity runs. Run titles look like: | ||
| # "<csv_name or SHA> · mi355, mi300, mi200" | ||
| # Once any parity run exists for a SHA, we do not dispatch another | ||
| # report for that SHA. This keeps the dashboard to one report per | ||
| # upstream commit. | ||
| EXISTING=$(gh run list \ | ||
| --repo "$GITHUB_REPOSITORY" \ | ||
| --workflow parity.yml \ | ||
| --limit 1000 \ | ||
| --json displayTitle 2>/dev/null || echo '[]') | ||
|
|
||
| sha_already_dispatched() { | ||
| local sha="$1" | ||
| echo "$EXISTING" | jq -e --arg sha "$sha" \ | ||
| 'any(.[]; .displayTitle | contains($sha))' >/dev/null |
There was a problem hiding this comment.
Since the same parity.yml can be triggered outside of the auto-parity, we should narrow this down to consider only the auto-parity-triggered runs of parity.yml. In addition to SHA, one more "variable" is the arch list. For simplicity, let us get rid of the arch list workflow input entirely in auto-parity, so that we don't have to worry about whether a particular run covered all the archs we wanted.
| # Pull recent parity runs. Run titles look like: | ||
| # "<csv_name or SHA> · mi355, mi300, mi200" | ||
| # Once any parity run exists for a SHA, we do not dispatch another | ||
| # report for that SHA. This keeps the dashboard to one report per | ||
| # upstream commit. | ||
| EXISTING=$(gh run list \ | ||
| --repo "$GITHUB_REPOSITORY" \ | ||
| --workflow parity.yml \ | ||
| --limit 1000 \ | ||
| --json displayTitle 2>/dev/null || echo '[]') | ||
|
|
||
| sha_already_dispatched() { | ||
| local sha="$1" | ||
| echo "$EXISTING" | jq -e --arg sha "$sha" \ | ||
| 'any(.[]; .displayTitle | contains($sha))' >/dev/null |
There was a problem hiding this comment.
That being said, I do prefer the suggested version by Copilot review above, since it gets rid of the magic number of 1000 and seems to make the window logic more robust by using the right API parameters.
|
Jenkins build for 1375a6adeea743262fe99f276cb8afb5c511af86 commit finished as NOT_BUILT |
|
Addressed the latest review feedback from Jithun:
Validation:
|
|
Jenkins build for 1375a6adeea743262fe99f276cb8afb5c511af86 commit finished as FAILURE |
Add a scheduled scanner that dispatches one parity report per ready upstream PyTorch main commit, with PR dry-runs to validate readiness without creating reports.
Scope auto parity to completed trunk mi355 default reports, remove user-facing regex inputs, and use paginated workflow APIs for candidate and dedupe windows.
Fetch completed trunk runs page by page only until the configured unique SHA limit is reached, avoiding full workflow-history scans.
ethanwee1
force-pushed
the
ethanwee/parity-auto-every-commit
branch
from
1375a6a to
a2b268a
Compare
|
Jenkins build for d7ff112ed2257d50c0031bdd24d25675b3f89e52 commit finished as NOT_BUILT |
Stop passing csv_name from parity-auto so auto-dispatched parity reports use the same output names as direct parity.yml runs.
Restore the auto parity ready-arch dispatch behavior while preserving parity.yml's default CSV naming by omitting csv_name.
|
Jenkins build for d7ff112ed2257d50c0031bdd24d25675b3f89e52 commit finished as FAILURE |
…efix Re-introduce the "autoparity-" marker on parity runs dispatched by parity-auto.yml so they are visually distinguishable in the Actions list and the title-based dedup keeps working - without reusing csv_name, which also drives the output/CSV filenames. Adds a dedicated boolean parity.yml input `auto_triggered` that only prefixes the run name with "autoparity-" (leaving csv_name and all output filenames untouched), and has parity-auto.yml pass `-f auto_triggered=true` on dispatch.
|
Jenkins build for 9363a9ff67078a129173686393c3c75d05a8105d commit finished as NOT_BUILT |
The every-commit auto-trigger should only cover what trunk.yml runs on
every main push: the mi355 (gfx950) ROCm test shards that ride along in
trunk.yml, compared against trunk's CUDA test shards. The other ROCm
arches (mi300, mi200, navi31, nightly) come from separate periodic
workflows that don't run on every commit, so including them produced
partial/late reports.
Reduce the defaults to mi355 only and restrict its workflow regex to
trunk.yml:
archs -> "mi355"
arch_jobname_regex_map -> {"mi355": "rocm.*mi355.*/ test (default|distributed|inductor),"}
arch_workflow_regex_map-> {"mi355": "(^|/)trunk.yml$"}
The CUDA gate is unchanged. The maps remain overridable inputs for manual
multi-arch dispatches.
|
Jenkins build for 8fa2be633504300696fbb45ac85f5bc53e212b7e commit finished as NOT_BUILT |
Add parity_job_config.json (shared with the download_testlogs PR #3278) and have parity-auto.yml read the ROCm check-run/workflow regexes and the CUDA check-run regex from it over the contents API at GITHUB_SHA, instead of hardcoding them. The arch_*_regex_map inputs become optional overrides (blank = use the config). One source of truth for job-name matching. parity.yml: readability only - a top-of-file pipeline overview, an explanation of the dense run-name precedence, and comments on the artifact-prefix logic and the download_testlogs flag builder. No behaviour change. Note: parity_job_config.json is also added in #3278; merge that first (or rebase) to avoid an add/add conflict.
|
Jenkins build for 608493bf99fbfdecabc97e03bc58f68aae8707a1 commit finished as NOT_BUILT |
Addresses review feedback: replace the per-line echo runs in the config-summary and step-summary blocks with single printf '%s\n' calls so the printing logic is a few lines instead of many.
|
Jenkins build for 608493bf99fbfdecabc97e03bc58f68aae8707a1 commit finished as FAILURE |
Summary
pytorch/pytorchmaintrunk.ymlpushes and dispatchesparity.ymlonce per ready upstream SHA.pull_requestdry-run path with a smaller scan window to validate the scanner without creating parity reports from PR CI.How it works
pytorch/pytorchtrunk.ymlpush runs onmain. Those trunk runs provide the candidate upstream SHAs to evaluate.ROCm/pytorchparity.ymlrun titles. If any existing parity run already contains that SHA, the SHA is skipped so we keep one report per upstream commit.mainbranch of pytorch/pytorch in any 10-minute intervalstatus=completed. It also waits for the CUDA default, distributed, and inductor check-runs consumed by parity.mem_leak_checkandrerun_disabled_testsare ignored because the parity report does not consume them.parity.ymlwith the ready arch list and a CSV prefix containing the upstream SHA, for exampleautoparity-YYYYMMDD-<sha>.dry_run=true, so they exercise the scanner and log would-be dispatches without creating reports. Scheduled and manually dispatched runs can create real parity reports.Test plan
yaml.BaseLoaderandbash -n.dry_run=false, scanned 20 recent upstream trunk runs, skipped SHAs with pending parity check-runs, dispatched 5 ready SHAs, and stopped atmax_dispatches=5.d76e83ef/mi355: https://github.com/ROCm/pytorch/actions/runs/26041518406457e1890/mi355: https://github.com/ROCm/pytorch/actions/runs/2604152899660f38508/mi355: https://github.com/ROCm/pytorch/actions/runs/26041541647d1d96569/mi355: https://github.com/ROCm/pytorch/actions/runs/260415518546e3cf2e4/mi355, mi300, mi200: https://github.com/ROCm/pytorch/actions/runs/26041618237Dispatch cadence note
max_dispatches=5only to avoid flooding ROCm/pytorch during manual testing.max_dispatches=50,max_commits=200, andmax_age_hours=72unless manually overridden.