[POC TEST] Canary Report - 0-48-3

[raissanjay]

Canary Report - POC Test Cycle 0-48-3

Stable baseline: v0.48.3
Canary version: v0.48.3
Sample window: 2026-05-13T07:28:30Z -> 2026-05-13T08:56:24Z

---

Volume

Cohort	Sessions	Users
stable	5	4
canary	11	2

---

Summary - rate metrics

Metric	Stable	Canary	Delta	Status
Errors / session	0.000	0.000	0.0%	OK
Sessions with errors	0.0%	0.0%	+0.0 pp	OK
cards_rendered fired	200.0%	200.0%	+0.0 pp	OK
target_missing fired	0.0%	0.0%	+0.0 pp	OK
fetch_fail fired	0.0%	0.0%	+0.0 pp	OK

---

Assertions - per-session pass rates

Check	Stable	Canary	Delta	Status
Container present	100.0%	100.0%	+0.0 pp	OK
Container has children	100.0%	100.0%	+0.0 pp	OK
All cards visible	100.0%	100.0%	+0.0 pp	OK
Filter panel present	100.0%	100.0%	+0.0 pp	OK
Sort present	100.0%	100.0%	+0.0 pp	OK
Search present	40.0%	88.9%	+48.9 pp	OK
Paginator present	0.0%	0.0%	+0.0 pp	OK
Load-more present	100.0%	100.0%	+0.0 pp	OK
Loader cleared	100.0%	100.0%	+0.0 pp	OK
No error UI shown	100.0%	100.0%	+0.0 pp	OK

---

Top issues - canary cohort

No errors detected in canary cohort.

---

Verdict

PROMOTE OK - no threshold breaches.

Sample size

• 11 canary sessions / 2 distinct users
• 5 stable sessions / 4 distinct users

Review the data above and form a verdict.

cc @sanrai

[cmiqueo]

(AI Generated) Structural Feedback

Sample Size Context: You have a significant imbalance in the current report (11 canary sessions for 2 users vs. 5 stable sessions for 4 users).

Recommendation: Add a "Statistical Significance" flag or a warning icon next to the Volume table if the user count is below a certain threshold (e.g., <50 users). This prevents teams from trusting a "PROMOTE OK" verdict based on only two people's experiences.

Metric Grouping: The "Summary" and "Assertions" tables overlap slightly in purpose.

Recommendation: Consolidate these into a single "Health Dashboard" but use Categorical Headers (e.g., Stability, Engagement, UI Integrity) to make it scannable for stakeholders who aren't developers.

Baseline Visibility: Currently, it says Stable baseline: v0.48.3 and Canary version: v0.48.3.

Recommendation: If the versions are identical (as in this POC), the report should highlight that this is a "Baseline Validation Run" rather than a comparison.

🚀 Recommended Functionality to Add

1. Cumulative Distribution Function (CDF) for Performance:
   "Rate metrics" tell you if things broke, but they don't tell you if they got slower. Adding a row for Time to First Card (TTFC) or LCP (Largest Contentful Paint) would be vital for catch performance regressions that don't throw errors.

2. Search Assertion Refinement:
   I noticed the Search present delta is +48.9 pp (Canary is actually performing "better" than stable).

Change: Instead of just "Present," add a functional check like "Search returns results." A 40% pass rate on the stable version suggests the assertion itself might be flaky or the environment isn't consistent.

3. Environment Metadata:
   Add a small "Environment" badge (e.g., Stage, Prod, Local-Agent). This is especially helpful when you're running tests through the Go-based agent we've been working on, so you know exactly which ingestion pipeline handled the data.

4. Deep Links to Logs:
   The "Top issues" section is currently empty. When an error is detected, the report should provide a direct markdown link to the specific log trace in your observability tool (Splunk, New Relic, etc.).

✂️ What to Remove or Change
Remove "Paginator present": Since it’s at 0.0% for both, if a feature isn't expected for a specific page (like a "load-more" vs "pagination" conflict), it should be hidden to reduce noise.

Change the Verdict Logic: The verdict "PROMOTE OK" is currently based on "no threshold breaches." I’d suggest adding a "Manual Review Required" status if the sample size is as low as it is in this report, even if the metrics look green.