[benchmarking] Add ping_users_on_failure toggle, disable in nightly config

[rlratzel]

Summary

Adds a sink-level boolean ping_users_on_failure (default true) to the
Slack sink, and disables it in the nightly benchmark config to silence
@-mentions while the test suite is being stabilized.

Why

When a benchmark entry fails, the Slack sink currently @-mentions every
user in the entry's per-entry ping_on_failure list — even when failures
are caused by known transient infrastructure or work-in-progress issues
rather than ownership-relevant regressions. The author of each affected
entry has agreed to temporarily silence pings until the suite is back to a
clean state.

The straightforward but disruptive way to do this would be to edit every
entry's sink_data.ping_on_failure list. Instead, this PR adds a
sink-level toggle so the per-entry lists are preserved and can be honored
again by flipping a single line (or removing it).

Changes

1. [benchmarking] Add ping_users_on_failure toggle to SlackSink (c210f05)

• benchmarking/runner/sinks/slack_sink.py:
  • SlackSink.__init__ reads sink_config.get("ping_users_on_failure", True).
  • register_benchmark_entry_finished gates the existing per-entry ping_on_failure lookup on the new flag.
• Default True so existing pipelines see no behavior change.

The new key is named ping_users_on_failure rather than overloading the
existing per-entry ping_on_failure — the latter is a list of Slack user
IDs, while this is a boolean. Same name with different types in different
config scopes would invite confusion and silent misconfiguration.

2. [benchmarking] Disable Slack @-mentions on failure in nightly config (da80d7a)

• benchmarking/nightly-benchmark.yaml: sets ping_users_on_failure: false
  on the slack sink block, with an inline comment explaining how to revert.

Config example

sinks:
  - name: slack
    enabled: true
    ping_users_on_failure: false   # ← new, default true
    ...

entries:
  - name: my_benchmark
    sink_data:
      - name: slack
        ping_on_failure:             # existing per-entry list, unchanged
          - U022P8CDX40

Reverting

To restore @-mentions, change ping_users_on_failure: false back to
true (or remove the line) in nightly-benchmark.yaml. No other edits
are needed — the per-entry ping_on_failure lists were not touched.

Test plan

• Pre-commit (ruff, ruff-format, signoff) clean
• Trigger a curator nightly run with at least one entry expected to
  fail (e.g. against a known-failing test from
  Curator PR #2035
  remaining failures) and confirm no @-mention appears in the
  Slack thread for the failed entry
• Flip ping_users_on_failure: true locally, re-run, confirm pings
  reappear (verifies default-true path)
• Run an unmodified config without the new key; confirm behavior
  matches current production (verifies the default-True read path)

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR adds a sink-level ping_users_on_failure boolean to SlackSink (defaulting to true) and sets it to false in the nightly benchmark config to temporarily suppress @-mentions while the test suite is being stabilized. The per-entry ping_on_failure user lists are preserved untouched, so pings can be restored with a one-line change.

• slack_sink.py: reads the new flag in __init__ and short-circuits the per-entry ping lookup in register_benchmark_entry_finished — logic is correct and the default=True path is fully backward-compatible.
• nightly-benchmark.yaml: adds ping_users_on_failure: false with an inline comment explaining how to revert.

Confidence Score: 5/5

Safe to merge — the new flag defaults to true, so all existing configs behave identically, and the only behavioral change is the opt-in false set in nightly-benchmark.yaml.

The change is minimal and self-contained: one new config key read in __init__, one conditional rewrite in register_benchmark_entry_finished, and one YAML line. The default True path exactly reproduces the old one-liner it replaced. The nightly config change is explicitly temporary and fully reversible without touching any per-entry data.

No files require special attention.

Important Files Changed

Filename	Overview
benchmarking/runner/sinks/slack_sink.py	Adds ping_users_on_failure bool flag to SlackSink.__init__ and gates per-entry ping lookups on it in register_benchmark_entry_finished; logic is correct and defaults preserve existing behavior.
benchmarking/nightly-benchmark.yaml	Adds ping_users_on_failure: false to the Slack sink block with a clear revert comment; change is isolated and reversible.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[register_benchmark_entry_finished] --> B{result_dict success?}
    B -- Yes --> C[pings = empty list]
    B -- No --> D{self.ping_users_on_failure?}
    D -- False\n sink-level toggle --> C
    D -- True\n default --> E[pings = sink_data.get ping_on_failure]
    C --> F[Post Slack message\nno @-mentions]
    E --> G[Post Slack message\nwith @-mentions]

Loading

_{Reviews (1): Last reviewed commit: "[benchmarking] Disable Slack @-mentions ..." | Re-trigger Greptile}