Skip to content

feat(ci): label fresh PRs waiting-for-review once they clear the bar - #4179

Merged
dhruv0811 merged 2 commits into
mainfrom
dhruvgupta/wfr-fresh-prs
Aug 5, 2026
Merged

feat(ci): label fresh PRs waiting-for-review once they clear the bar#4179
dhruv0811 merged 2 commits into
mainfrom
dhruvgupta/wfr-fresh-prs

Conversation

@dhruv0811

Copy link
Copy Markdown
Contributor

Related issue

Part of OMNI-2214 / OMNI-2315. Closes the last gap in the review-state machine;
this PR declares Test / CI below, which is one of the exempt types.

Summary

waiting-for-review had exactly one entrance: hand_off_to_reviewer, which fires
when an author replies to feedback on a PR already carrying waiting-on-author.

A PR nobody had touched yet therefore sat in neither state. That is why 478 of
479 currently-open PRs carry no review-state label, and why the label cannot yet be
used as the review queue it was built to be. A fresh PR that references an issue
should be labelled ready, and today nothing does that.

This adds a sweep step that applies the label to PRs clearing the bar.

The bar today is just "references an issue." It reuses pr-issue-link.js's
resolution (closing links, sidebar links, and Part of / Related to / Towards
/ Refs pointing at a real issue), so the gate and the nudge can never disagree
about what counts. It is designed to rise: CI green, demo present, Polly clean each
become one more predicate in belowBar, and nothing else changes.

Never applied when:

  • the PR is a draft (the author is telling us it is not ready)
  • waiting-on-author is set (the ball is in the author's court, and applying both
    would break the mutual exclusion the pair relies on)
  • the label is already there (idempotent)
  • a human removed it before. A maintainer who takes the label off is saying
    "not ready"; an hourly sweep that reapplies it would be arguing with them. An
    unrelated label removal is ignored.

Forward-only, sharing EFFECTIVE_FROM with the issue-link check: labelling 478
backlog PRs in one sweep would bury the signal the label exists to create.

Ships dry-run (ENFORCE: "false"), consistent with how the nudge shipped.

Test Plan

Demo

N/A. CI label mechanics, no user-visible UI.

Type of change

  • Bug fix
  • Feature
  • UI / frontend change
  • Refactor / chore
  • Docs
  • Test / CI
  • Breaking change

Test coverage

  • Unit tests added or updated
  • Integration / E2E tests added or updated
  • Manual verification completed
  • Not applicable

Coverage notes

Unit tests cover every predicate and exclusion. Manual verification was running the
shipped script against live GitHub with the label write rigged to throw, confirming
both the verdict set and that nothing was written.

Follow-up

Once this is enforcing, CONTRIBUTING can state that reviewers work the
waiting-for-review queue. It deliberately does not say that today, because the
label was not yet applied to fresh PRs and the doc would have read as "we will
never review your PR."

`waiting-for-review` had exactly one entrance: the handoff that fires when an
author replies to feedback. A PR nobody had touched yet sat in neither state, so
478 of 479 open PRs carry no review-state label and the label cannot yet be used
as a review queue.

A new sweep step applies it to PRs that clear the bar. The bar today is just
"references an issue", reusing pr-issue-link.js's resolution so the gate and the
nudge can never disagree about what counts. It is meant to rise: CI green, demo
present, Polly clean each become a predicate in `belowBar`.

Never applied to a draft, to a PR already carrying `waiting-on-author` (which
would break the mutual exclusion the pair relies on), or to a PR whose label a
human removed before, since a sweep that reapplies it hourly would be arguing
with the maintainer who took it off. Forward-only, sharing the issue-link
effective date, because labelling the whole backlog at once would bury the signal.

Ships dry-run. Verified against production with the label write rigged to throw:
26 PRs in the window, 4 ready, 20 below bar, 2 drafts skipped, no writes attempted.

Co-authored-by: Isaac
Signed-off-by: Dhruv Gupta <dhruv.gupta@databricks.com>
Copilot AI lite review requested due to automatic review settings August 5, 2026 22:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added the size/L Pull request size: L label Aug 5, 2026
@dhruv0811

Copy link
Copy Markdown
Contributor Author

/review

@omnigent-ci

omnigent-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

removedBefore ignores the actor, so the bot's own mutual-exclusion removal permanently disqualifies a PR. In ready-for-review.js:

function removedBefore(pr) {
  const events = pr.timelineItems?.nodes ?? [];
  return events.some((e) => e?.label?.name === REVIEW_LABEL);
}

The GraphQL query selects actor { login } on each UnlabeledEvent, and the doc comment states the intent explicitly — "a maintainer who takes it off is saying 'not ready'", "an unrelated label removal is ignored". But the function only matches on label name; it never inspects the actor. That matters because this repo's own automation removes exactly this label: .github/scripts/waiting_on_author.py::clear_review_label_on_waiting deletes waiting-for-review whenever waiting-on-author is applied (the mutual-exclusion pair). That deletion is performed by github-actions[bot] and is recorded as an UnlabeledEvent for waiting-for-review.

Reachable sequence:

  1. Sweep labels a fresh PR waiting-for-review.
  2. A maintainer engages → waiting-on-author applied → bot removes waiting-for-review (UnlabeledEvent, actor = bot).
  3. The waiting-on-author state is later cleared without an author-activity handoff (e.g. a maintainer removes it manually).
  4. PR is now in neither state — exactly the gap this PR exists to close — but removedBefore returns true forever, so the sweep will never re-label it.

So the bot's routine state transition is misread as a human "not ready" signal, silently and permanently excluding the PR from the very queue this change builds. The fix is to filter to human actors (exclude [bot] logins / the workflow bot) before treating a removal as intentional — which is what the query already fetches the actor for. Note the test suite (cases #17/#18) only exercises human-attributed removals, so this path is untested.

Security vulnerabilities

None. Actions are pinned by SHA; the sweep runs from the trusted default-branch checkout (never PR head), the test workflow uses persist-credentials: false with contents: read only, and no secrets are read or emitted. Label writes require the issues: write already granted to the demo-check job.

Non-blocking notes

  • Dead field / self-documenting gap. actor { login } is queried but never referenced anywhere. Once the blocking issue above is addressed it becomes live; if you deliberately keep name-only matching, drop the actor selection so the code and comment stop implying an actor check that doesn't happen.
  • Partial sweep on write failure. A thrown addLabels (non-429) aborts the loop mid-run, labelling a prefix of the ready set and skipping the rest. Harmless given idempotency and hourly re-runs, but worth a per-PR try/catch so one failure doesn't stop the sweep (mirrors the resilience close_stale_waiting_prs has in the Python script).
  • referencesIssue resolves cross-repo owner/repo#N refs against the local repo (same as pr-issue-link.js), so a Refs other/repo#5 could match a coincidental local issue Sync 🌊: 48 upstream changes #5. Consistent with the existing nudge, so not a regression — flagging only for awareness.

Summary

Well-scoped, well-tested addition that correctly reuses pr-issue-link.js resolution, honors the draft / waiting-on-author / idempotency exclusions, and ships safely in dry-run behind pinned actions with least-privilege permissions. The one real problem is removedBefore: it claims to respect only human removals but matches any removal, and this repo's mutual-exclusion bot removes precisely this label — so a common lifecycle transition can permanently disqualify a PR from the queue the feature is meant to populate. Filter the removal check to human actors (the actor is already fetched) and add a bot-removal test case; the rest is ready.


Automated review by Polly · workflow run

removedBefore matched any removal of waiting-for-review, ignoring the actor the
query already fetched. But waiting_on_author.py removes that label itself on every
waiting-on-author transition, since the two are mutually exclusive, so the bot's
own routine state change was read as a maintainer saying "not ready".

The effect was permanent: a PR that had been through one review round trip and then
ended up in neither state, which is exactly the gap this gate exists to close, would
never be re-labelled. Confirmed on a real PR from earlier today whose timeline
records "unlabeled waiting-for-review by github-actions[bot]".

Rename to removedByHuman and filter out [bot] actors. A missing actor fails toward
eligible, since a removal we cannot attribute is not evidence of intent.

Also make the label write per-PR so one failure no longer abandons the rest of the
sweep, matching the resilience close_stale_waiting_prs already has.

Co-authored-by: Isaac
Signed-off-by: Dhruv Gupta <dhruv.gupta@databricks.com>
Copilot AI review requested due to automatic review settings August 5, 2026 22:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot added size/XL Pull request size: XL and removed size/L Pull request size: L labels Aug 5, 2026
@dhruv0811

Copy link
Copy Markdown
Contributor Author

/review

2 similar comments
@dhruv0811

Copy link
Copy Markdown
Contributor Author

/review

@dhruv0811

Copy link
Copy Markdown
Contributor Author

/review

@omnigent-ci

omnigent-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

None. The logic is correct and the test suite passes. I applied the patch locally and ran node .github/workflows/ready-for-review.test.js — all assertions pass. I verified the specifics that matter most:

  • The new Ready-for-review gate step is appended to the demo-check job, which already declares issues: write / pull-requests: write, so addLabels has the permission it needs. if: always() keeps it independent of the earlier steps.
  • The demo-check job checks out sparse-checkout: .github on the default branch, so both ready-for-review.js and its require("./pr-issue-link.js") dependency are present, and no PR-authored code runs.
  • removedByHuman correctly excludes the github-actions[bot] self-removal that waiting_on_author.py performs on every waiting-on-author transition (REVIEW_LABEL not in labels → remove), which would otherwise permanently disqualify most PRs. The endsWith("[bot]") actor check is the right guard and is directly exercised by the #181 / #182 tests.
  • Fail-closed behavior is consistent: a LINK_QUERY error returns false (below bar, no label) and an unresolvable tracking-reference lookup is swallowed and skipped — matching the nudge's contract so the gate and nudge can't disagree.
  • Rate-limit 429s re-throw out of the per-PR addLabels catch and are handled by the top-level graceful exit.

Security vulnerabilities

None. The new ready-for-review-test.yml runs the PR's own code but uses the safe pattern: pull_request (not pull_request_target), permissions: contents: read only, persist-credentials: false, no secrets, and the test is fully mocked/offline. The label-writing script only ever runs from the trusted default-branch checkout in the scheduled sweep. No injection, secret exposure, or boundary weakening.

Non-blocking notes

  • 24h rolling window can miss late-linked PRs. The sweep searches is:open created:>cutoff with a 24h window (shared with the nudge). A PR that opens without an issue reference and then adds one on day 2+ has already aged out of the window, so it will never enter the waiting-for-review queue via this sweep — its only entrance remains hand_off_to_reviewer. Since the stated goal is "a queue of reviewable PRs," this is a small gap worth being aware of; it's consistent with the existing nudge design, so it's fine to defer.
  • removedByHuman is a permanent disqualification. As the code comments acknowledge, a single historical human removal permanently opts a PR out of the sweep even after substantial revision. Intended, but worth noting for when the bar rises (CI-green/demo/Polly predicates) — a stale "not ready" signal will suppress re-labelling.
  • timelineItems(last: 50, itemTypes: [UNLABELED_EVENT]) is uncapped-safe only up to 50 events. A very churny PR with >50 unlabel events could push an older human removal out of view. Extremely unlikely on fresh PRs; no action needed.

Summary

A tight, well-scoped CI addition that closes the fresh-PR entrance to waiting-for-review. It correctly reuses the nudge's issue-resolution logic, handles the bot-vs-human removal distinction that would otherwise poison the gate, fails closed on unverifiable lookups, ships dry-run by default, and is backed by a thorough 12-case offline test plus a safe PR-triggered test workflow. No blocking or security issues. Ready to merge once maintainers are comfortable flipping ENFORCE to "true" after reviewing the dry-run summary.


Automated review by Polly · workflow run

@dhruv0811
dhruv0811 merged commit d7701e5 into main Aug 5, 2026
64 checks passed
@dhruv0811
dhruv0811 deleted the dhruvgupta/wfr-fresh-prs branch August 5, 2026 22:52
@github-actions github-actions Bot added the no-doc-update Merged PR does not need a docs update label Aug 5, 2026
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🏷️ Doc impact: no-doc-update

Adds a CI-only GitHub Actions "ready-for-review" labeling gate plus its workflow and unit tests, an internal maintainer tooling change with no user-facing surface or integration affected.

Auto-classified on merge. Set the label manually before merging to override. · run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-doc-update Merged PR does not need a docs update size/XL Pull request size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants