From ffea47f62f8f0df331cd8b277a3ad71736c02e90 Mon Sep 17 00:00:00 2001 From: Chai Bot Date: Sat, 1 Aug 2026 18:40:16 +0000 Subject: [PATCH 1/2] fix: call schedule_followup before send_response in step 7 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit send_response() is a hard turn-terminator — no tool calls execute after it. The previous ordering (send_response first, then schedule_followup) meant the remediation follow-up was never scheduled, breaking the daily health → remediation chain. Reorder to: schedule_followup() → send_response(). Also fix the same ordering bug in the chained PR→Jira remediation follow-up. Co-Authored-By: Claude Opus 4.6 --- .chai-bot/rosa_ci_daily_health_report.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/.chai-bot/rosa_ci_daily_health_report.md b/.chai-bot/rosa_ci_daily_health_report.md index d774901..fd2e31c 100644 --- a/.chai-bot/rosa_ci_daily_health_report.md +++ b/.chai-bot/rosa_ci_daily_health_report.md @@ -200,20 +200,19 @@ categories: ### 7. Deliver response and schedule remediation follow-up -1. Call `send_response()` to deliver the summary (and threaded replies if applicable). **`send_response()` ends your current turn — no tool calls after it.** + **All-green path** (every category >= 80%): Call `send_response()` to deliver the summary. Do **not** schedule a remediation follow-up — there is nothing to remediate. The artifact is still written so PR shepherding can happen if triggered manually. -2. **After `send_response()`**, call `schedule_followup` with a 2-minute delay. The follow-up fires **in the same thread** as the health report, so threading is automatic — no thread_ts or channel_id needed. + **Failures present** (any category < 80%): - **All-green path** (every category >= 80%): Write the handoff artifact (step 6) and call `send_response()`. Do **not** schedule a remediation follow-up — there is nothing to remediate. The artifact is still written so PR shepherding can happen if triggered manually. - - **Failures present** (any category < 80%): After `send_response()`, schedule the follow-up with this description (replace `` with the actual fork repo path from step 6.1 and `` with the commit SHA from step 6.5): + 1. **First**, call `schedule_followup` with a 2-minute delay. The follow-up fires **in the same thread** as the health report, so threading is automatic — no thread_ts or channel_id needed. Use this description (replace `` with the actual fork repo path from step 6.1 and `` with the commit SHA from step 6.5): + 2. **Then**, call `send_response()` to deliver the summary (and threaded replies). **`send_response()` ends your current turn — no tool calls after it.** > PR Remediation follow-up. You are continuing the daily health report thread with automated remediation. > > 1. Call `priv_scm_ensure_fork("github.com", "openshift-online/rosa-e2e")` to resolve the fork path. > 2. Read the handoff artifact using `github_file_content(repo="", path=".chai-bot/reports/daily_health_latest.yaml", ref="")`. Parse the YAML. Verify `report_date` is today — if stale, call `no_action_required()`. > 3. Read the remediation instructions using `github_file_content(repo="openshift-online/rosa-e2e", path=".chai-bot/rosa_ci_daily_remediation.md")`. Follow the **"## PR Remediation"** section (auto-fix PRs and PR shepherding). - > 4. **Action branch** (PRs were opened, shepherded, or closed): Compose a summary of all PR actions taken and post it using `send_response()`. After `send_response()`, call `schedule_followup` with a 2-minute delay for Jira remediation (use the description in step 6). + > 4. **Action branch** (PRs were opened, shepherded, or closed): Compose a summary of all PR actions taken. Call `schedule_followup` with a 2-minute delay for Jira remediation (use the description in step 6), then call `send_response()` to post the summary. > 5. **No-action branch** (no fixable failures, no open PRs to shepherd): Schedule the Jira follow-up directly (persistent failures may still need tickets even if no PRs are warranted), then call `no_action_required()`. Do NOT post an empty summary via `send_response()`. > 6. Jira follow-up description (used by both branches): > "Jira Remediation follow-up. You are continuing the daily health report thread with Jira ticket creation for persistent failures. (1) Call priv_scm_ensure_fork to resolve the fork. (2) Read the handoff artifact from at .chai-bot/reports/daily_health_latest.yaml using ref=. Verify report_date is today. (3) Read the remediation instructions from openshift-online/rosa-e2e at .chai-bot/rosa_ci_daily_remediation.md. Follow the '## Jira Remediation' section. (4) Compose a summary of Jira actions taken and post it using send_response(). If no Jira actions are needed, call no_action_required()." @@ -226,6 +225,6 @@ categories: - Before sending: if any category is below 80%, verify your response content contains `---THREAD_DETAILS---` followed by at least one threaded reply section. If these delimiters are missing, your threaded replies will not be posted — go back to step 5. - Always write the handoff artifact (step 6) before calling `send_response()`, even if all categories are green. - Only ONE pending follow-up per thread at a time. Do not schedule multiple follow-ups from the same turn. -- `send_response()` ends the turn immediately — no tool calls after it except `schedule_followup`. +- `send_response()` ends the turn immediately — no tool calls after it. Always call `schedule_followup` before `send_response()` when a follow-up is needed. - On the all-green path (all categories >= 80%), do NOT schedule a remediation follow-up. From 9173288d7f3a2ae0cf1c62b2c13abf8702ca888d Mon Sep 17 00:00:00 2001 From: Chai Bot Date: Sat, 1 Aug 2026 18:45:57 +0000 Subject: [PATCH 2/2] fix: add PR-first philosophy and expand auto-fix patterns in remediation - Add "Guiding principle: PR-first remediation" section establishing that PRs should be the default action, with Jira as a fallback - Expand auto-fix PRs section with fixable patterns for rosa-e2e test code, ROSA CLI tests, CI workflow/config, SRE operator tests, and OCM FVT tests - Update Jira section to explicitly position tickets as the fallback when PR-based fixes are not feasible - Fix schedule_followup/send_response ordering constraint (same bug as health report fix) Co-Authored-By: Claude Opus 4.6 --- .chai-bot/rosa_ci_daily_remediation.md | 53 ++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 3 deletions(-) diff --git a/.chai-bot/rosa_ci_daily_remediation.md b/.chai-bot/rosa_ci_daily_remediation.md index a44905c..f802422 100644 --- a/.chai-bot/rosa_ci_daily_remediation.md +++ b/.chai-bot/rosa_ci_daily_remediation.md @@ -4,6 +4,15 @@ > > The follow-up description tells you which section to execute and how to read the handoff artifact. You do not need to resolve threading or read the artifact independently — that context is provided in the follow-up prompt. +## Guiding principle: PR-first remediation + +**Attempt to directly fix failures via PRs whenever possible.** The fastest path to green CI is a code change, not a ticket. For every persistent failure, first ask: "Can I open a PR to fix this right now?" Only fall back to Jira when: +- The fix requires changes to repos outside the allowed list (see constraints below) +- The failure requires deep domain investigation or design changes that cannot be safely automated +- The root cause is in an upstream dependency with no clear workaround + +This means the PR Remediation phase should be aggressive — go beyond skip-list updates to fix test bugs, adjust timeouts, update CLI flags, correct CI configs, and patch flaky assertions. A merged PR closes the loop; a Jira ticket opens one. + ## PR Remediation This section covers automated PR fixes for pattern-matched failures and shepherding of existing `[ci-fix]` PRs. @@ -28,6 +37,42 @@ If a conformance test (HCP or Classic STS) is failing persistently (3+ consecuti 6. Open a PR with title `[ci-fix] Skip in (upstream OCP regression)` 7. PR description must link to the failing Prow job run(s) and reference the upstream OCP bug if identifiable +**Test code fixes in `openshift-online/rosa-e2e`:** + +If a rosa-e2e test is failing due to a bug in the test itself (not the product), fix it directly: +- Timeout adjustments: increase timeouts for operations that legitimately take longer in certain environments +- Flaky assertion fixes: stabilize assertions that race against async operations (e.g., add polling/retry, wait for conditions) +- Test environment setup: fix incorrect assumptions about cluster state, missing skip conditions for topology/access, or stale configuration references +- Label corrections: fix tests running on wrong topologies due to missing or incorrect platform labels + +**ROSA CLI test fixes in `openshift/rosa`:** + +If a ROSA CLI E2E test is failing due to CLI changes (not service-side issues): +- Version gate adjustments: update version checks when minimum supported versions change +- Flag/argument changes: update test commands when CLI flags are renamed, deprecated, or added +- Output format changes: update assertion patterns when CLI output format changes + +**CI workflow/config fixes in `openshift/release`:** + +Beyond conformance skip lists, fix CI infrastructure issues: +- Step registry updates: fix broken step references, update image tags, correct environment variable names +- Resource adjustments: increase memory/CPU limits for steps that are OOM-killed or throttled +- Timeout tuning: adjust step or overall job timeouts based on observed run durations +- Environment variable fixes: correct or add required env vars in workflow definitions + +**SRE operator test/config fixes:** + +For SRE operator repos listed in the allowed repos (see constraints): +- Test fixes: stabilize flaky operator tests, fix assertion logic, adjust timeouts +- Config corrections: fix test configuration that references stale or incorrect resources + +**OCM FVT test fixes in `service/ocm-backend-tests` (GitLab):** + +If cs-telemetry data confirms the failure is test-side (assertion errors, framework issues), not CS-side (API errors, timeouts): +- Fix test assertions that no longer match current API behavior +- Update test data or fixtures for changed API schemas +- Adjust test timeouts for operations that legitimately take longer + **Constraints:** - Maximum **5** auto-fix PRs per scheduled run - Allowed repos for fixes: @@ -100,9 +145,11 @@ After completing auto-fix PRs and PR shepherding, compose a summary of all PR ac This section covers Jira ticket creation for persistent non-fixable failures. -### Jira ticket creation (for non-fixable failures) +### Jira ticket creation (for failures not fixable via PR) + +Jira tickets are the **fallback**, not the default. Only create a ticket when a PR-based fix is genuinely not feasible — the failure requires design changes, upstream fixes in repos outside the allowed list, or deep investigation by domain experts that cannot be safely automated. -For persistent failures (3+ consecutive) where auto-fix PRs were not opened (the failure requires deeper investigation or a fix outside the allowed repos), create a Jira ticket so the owning team can investigate. +For persistent failures (3+ consecutive) where auto-fix PRs were not opened, create a Jira ticket so the owning team can investigate. **Skip fetch_error jobs:** Jobs with `failure_classification` of `"fetch_error"` represent data retrieval failures, not real test failures. Skip these entirely — do not create Jira tickets for them. @@ -170,4 +217,4 @@ Post this summary using `send_response()`. If no Jira actions were taken (no per - If the handoff artifact is missing or stale (not today's date), call `no_action_required()`. - Each follow-up gets its own token budget (fresh context). Re-read the artifact and these instructions from the repo at the start of each follow-up. - Only ONE pending follow-up per thread at a time. The PR remediation follow-up schedules the Jira follow-up; do not schedule both from the same turn. -- `send_response()` ends the turn immediately — no tool calls after it except `schedule_followup`. +- Call `schedule_followup` BEFORE `send_response()` — `send_response()` ends the turn immediately and no tool calls can execute after it.