ROSAENG-394: Update CI watcher docs with signal sources, triage process, and Slack workflows

[dustman9000]

Summary

Updates all five CI watcher docs (docs/ci-watcher/) to reflect current tooling and strengthen the triage process:

• Rename #wg-rosa-ci-enhancement to #wg-rosa-cicd throughout (channel was renamed)
• Add direct Slack channel links (channel IDs) to future-proof against name changes
• Add #rosa-prow-info as a real-time signal source (failures now tag @rosa-ci-watcher)
• Add chai-bot daily health report as the starting point, with detail on how /ci-triage complements it
• Add Triage Before Routing section with a 5-step checklist and mermaid flowchart
• Add shift start/end Slack Workflow for automated handovers (free-form, not repo-committed docs)
• Add ROSA Engineering Dashboard links (CI Health, Delivery)
• Add PagerDuty CI Watcher team link (PTYDD6M)
• Add stage environment health check and cluster ID / ServiceLogs capture to the daily workflow
• Add routing without triage as the first anti-pattern

Jira: https://redhat.atlassian.net/browse/ROSAENG-394

Test plan

• Review mermaid diagrams render correctly on GitHub
• Verify all external links resolve (PagerDuty, Sippy, Prow, rosa-eng-dashboard, Slack)

[openshift-ci-robot]

@dustman9000: This pull request references ROSAENG-394 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Updates all four CI watcher docs (docs/ci-watcher/) to reflect current tooling and strengthen the triage process:

• Add #rosa-prow-info channel as a real-time signal source (failures now tag @rosa-ci-watcher)
• Add chai-bot daily health report as the starting point, with detail on how /ci-triage complements it (chai-bot covers top 5 with detailed analysis, /ci-triage covers all jobs and produces Jiras + fix PRs)
• Add Triage Before Routing section with a 5-step checklist and mermaid diagram
• Add shift start/end Slack Workflow for automated handovers (free-form, not repo-committed docs)
• Add ROSA Engineering Dashboard links (CI Health, Delivery)
• Add PagerDuty CI Watcher team link (PTYDD6M)
• Add stage environment health check and cluster ID / ServiceLogs capture to the daily workflow
• Add routing without triage as the first anti-pattern

Jira: https://redhat.atlassian.net/browse/ROSAENG-394

Test plan

• Review mermaid diagrams render correctly on GitHub
• Verify all external links resolve (PagerDuty, Sippy, Prow, rosa-eng-dashboard)
• Confirm rosa-prow-info channel ID matches (C0AT31ERJLS)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dustman9000

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dustman9000]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Walkthrough

The CI Watcher documentation adds onboarding resources, investigation-before-routing requirements, an eight-step daily triage workflow, expanded health checks, and Slack Workflow-based shift handovers.

Changes

CI Watcher documentation

Layer / File(s)	Summary
Onboarding and resource references docs/ci-watcher/README.md, docs/ci-watcher/rotation-schedule.md	Onboarding now includes team membership and #rosa-prow-info checks. The resource list adds PagerDuty, CI health, delivery status, Prow notifications, and Chai-bot links.
Triage responsibilities and routing docs/ci-watcher/role-and-responsibilities.md	The role guidance requires log review, health checks, reproducibility checks, Jira context, contextual routing, and threaded status communication.
Daily runbook and shift handover docs/ci-watcher/runbook.md	The runbook adds an eight-step triage flow, Sippy and dashboard checks, expanded failure investigation, and Slack Workflow handovers.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• openshift-online/rosa-e2e#125: Shares the CI Watcher and Chai-bot health-report workflow.
• openshift-online/rosa-e2e#131: Shares CI Watcher Slack notifications and failure-routing procedures.
• openshift-online/rosa-e2e#155: Shares the ROSA Engineering Dashboard and Chai-bot reporting workflow.

Suggested labels: lgtm

Suggested reviewers: bmeng

🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Ai-Attribution	⚠️ Warning	The PR diff references Claude Code and AI output, but its sole commit has no Assisted-by or Generated-by trailer; no AI Co-Authored-By trailer is present.	Add a Red Hat Assisted-by or Generated-by trailer to the PR commit. Do not use Co-Authored-By for the AI tool.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
No-Weak-Crypto	✅ Passed	The PR changes only four Markdown documents; scans of added lines and current docs found no MD5, SHA1, DES, RC4, Blowfish, ECB, crypto implementation, or secret comparison.
Container-Privileges	✅ Passed	The PR changes only four Markdown documents; added lines contain no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or root-execution settings.
No-Sensitive-Data-In-Logs	✅ Passed	The PR changes only CI watcher documentation; it adds no logging code or sensitive literals, and its log guidance does not instruct users to print or publish secrets, PII, or customer data.
No-Hardcoded-Secrets	✅ Passed	The four changed Markdown files contain no credential URLs, private-key headers, secret assignments, bearer credentials, or base64 strings over 32 characters; PagerDuty IDs are resource identifiers.
No-Injection-Vectors	✅ Passed	The PR changes only four Markdown files; the patch contains no SQL concatenation or listed unsafe sinks, and repository search found no such constructs in the changed content.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the documentation updates, signal sources, triage process, and Slack workflows described in the pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/ci-watcher/README.md`:
- Around line 20-21: Update the Chai-bot health report entry to match the
weekday cadence documented in the runbook, replacing “daily” with “weekday”
while preserving the 14:30 UTC time, destination channel, and report details.

In `@docs/ci-watcher/role-and-responsibilities.md`:
- Around line 88-89: Use a single Slack Workflow handover artifact throughout
the CI watcher documentation: update the “Known issue?” step in
docs/ci-watcher/role-and-responsibilities.md lines 88-89 to search the Slack
Workflow handover thread, replace the handover-document reference at lines
101-101 with that artifact, and update docs/ci-watcher/runbook.md lines 109-112
to link the previous Slack Workflow thread or explicitly define the generated
document.

In `@docs/ci-watcher/runbook.md`:
- Around line 27-29: Update the “Step 1: Review Chai-Bot Daily Health Report”
schedule description to avoid equating 14:30 UTC with a single Pacific time
year-round. Keep 14:30 UTC as the canonical time and either remove the Pacific
conversion or document both PDT and PST offsets.
- Around line 56-58: Add the shell language identifier to the fenced command
block containing /ci-triage, changing the opening fence to specify shell while
preserving the command and closing fence.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 09655417-50f5-4169-9465-db6659b4cba8

📥 Commits

Reviewing files that changed from the base of the PR and between 703c54f and 2c6ec1f.

📒 Files selected for processing (4)

• docs/ci-watcher/README.md
• docs/ci-watcher/role-and-responsibilities.md
• docs/ci-watcher/rotation-schedule.md
• docs/ci-watcher/runbook.md

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Match the documented Chai-bot cadence.

This entry says daily, but docs/ci-watcher/runbook.md states that the report is posted every weekday at 14:30 UTC. Change this wording to weekday or state the weekday schedule.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/ci-watcher/README.md` around lines 20 - 21, Update the Chai-bot health
report entry to match the weekday cadence documented in the runbook, replacing
“daily” with “weekday” while preserving the 14:30 UTC time, destination channel,
and report details.