Optimize the ATOM CI Execution Time

[gyohuangxin]

ATOM CI Optimization

Current ATOM CI status

Estimated GPUs = one full workflow run.

Title	Runs On	Estimated GPUs / Run
Pre Checkin	PR, main	0
ATOM Test	PR, main	PR ~80 / main ~136
ATOM vLLM Test	PR only	~32
ATOM SGLang Test	PR only	~32
Atomesh Accuracy Validation	PR, main, nightly	~25
Nightly Docker Release	nightly, manual	~1
ATOM vLLM Nightly	nightly, manual	~205
ATOM SGLang Accuracy Validation	nightly, manual	~56-64
MMStar Accuracy	nightly, manual	~1
ATOM Benchmark	nightly, manual	~2,336
ATOM vLLM Benchmark	nightly, manual	~512
ATOM SGLang Benchmark	nightly, manual	~720
Atomesh Benchmark	PR, main, nightly, manual	~56
Atomesh Mocker Benchmark	PR, main, nightly, manual	0

Optimization plan

ATOM CI Optimization Ideas

• Tier PR CI

  • PRs run a default smoke set only:
    • Small models
    • Critical paths
    • 1-2 representative large models
  • Full ATOM Test runs only on:
    • Label trigger
    • main
  • Expected benefit:
    • Reduce PR GPU demand from ~80-136 GPUs to ~16-32 GPUs

• Select tests based on changed files

  • Docs/config/UI-only changes do not run GPU tests
  • vLLM-related changes run only ATOM vLLM Test
  • SGLang-related changes run only ATOM SGLang Test
  • Benchmark config changes run only benchmark dry-run or a small benchmark matrix

• Split benchmark coverage by day or group

  • Avoid running every model x scenario x concurrency combination in one nightly run
  • Rotate benchmark coverage by model or scenario:
    • Day 1: DeepSeek
    • Day 2: Kimi
    • Day 3: GLM
    • Day 4: gpt-oss
  • Keep a small always-on sentinel benchmark every day

• Reuse build, image, and artifacts

  • Produce shared artifacts once and reuse them across workflows:
    • aiter wheel
    • Docker image
    • Model predownload output
  • Avoid rebuilding or redownloading the same dependency in every GPU job

• Improve model cache and predownload service

  • Maintain a stable model cache on 8-GPU runners
  • Benchmark/accuracy jobs should validate cache hits before running
  • Avoid downloading large models on the critical path
  • Reduce time wasted on image pull and model download

[PerryZhang01]

Triggering PR builds on every code change is a waste of machine resources. We can change it to:

1. Trigger builds on every change only after the PR has been approved, since there are usually fewer changes after approval.
2. Allow manual triggering by commenting build on the PR when needed.
3. Run only lightweight checks, such as formatting checks, for each PR update.

[gyohuangxin]

Triggering PR builds on every code change is a waste of machine resources. We can change it to:

1. Trigger builds on every change only after the PR has been approved, since there are usually fewer changes after approval.
2. Allow manual triggering by commenting build on the PR when needed.
3. Run only lightweight checks, such as formatting checks, for each PR update.

Yes, this makes sense.

1. For the approval-based trigger, we can keep the existing pull_request workflow trigger, but add an approval gate before the
   expensive jobs. The workflow would still start on PR updates, but it would first check the PR review status. If the PR has not
   been approved yet, it would skip heavy jobs such as full builds, large tests, downstream tests, or benchmarks. Once the PR has
   been approved, any later code changes would pass the approval gate and trigger the full pipeline. This is relatively easy to
   implement and does not require changing the overall workflow structure too much.
2. For manual triggering, using labels would be easier to implement and maintain than parsing PR comments. For example, we can
   support labels like ci:full, ci:atom, or ci:benchmark to explicitly request heavier jobs when needed.
3. For every PR update before approval, we can keep only lightweight checks running, such as formatting checks and basic UT. The
   expensive build/benchmark jobs would run only after approval or when explicitly requested by label.

[gyohuangxin]

Test Plan

• Open a test PR that changes a non-doc file covered by ATOM heavy CI.

  • Expected: Pre Checkin runs automatically.
  • Expected: native ATOM / vLLM / SGLang heavy workflows stop at ci-gate before approval.

• Add ci:atom to the PR.

  • Expected: only ATOM Test passes ci-gate and continues after Pre Checkin passes.
  • Expected: vLLM and SGLang remain skipped unless separately requested.

• Add ci:vllm to the PR.

  • Expected: only ATOM vLLM Test passes ci-gate.

• Add ci:sglang to the PR.

  • Expected: only ATOM SGLang Test passes ci-gate.

• Add ci:full to the PR.

  • Expected: native ATOM, vLLM, and SGLang workflows all pass ci-gate.

• Submit an approval review.

  • Expected: pull_request_review triggers the gate immediately.
  • Expected: if final reviewDecision is APPROVED and Pre Checkin passed, heavy workflows continue.

• Submit a review comment or request changes.

  • Expected: workflow may start, but ci-gate skips.
  • Expected: it does not cancel an in-progress heavy run.

• Open or approve a docs-only PR.

  • Expected: ci-gate reports ignored paths only and heavy jobs stay skipped.

• Push another commit after the PR is approved.

  • Expected: heavy workflows pass ci-gate again and run after Pre Checkin passes.

• Open or reopen a PR.

  • Expected: PR welcome comment appears and lists ci:full, ci:atom, ci:vllm, and ci:sglang.