Skip to content

fix(detectors): preserve TriggerListDetector result alignment - #2000

Open
SarveshaKumarKS wants to merge 1 commit into
NVIDIA:mainfrom
SarveshaKumarKS:codex/fix-trigger-list-missing-notes
Open

fix(detectors): preserve TriggerListDetector result alignment#2000
SarveshaKumarKS wants to merge 1 commit into
NVIDIA:mainfrom
SarveshaKumarKS:codex/fix-trigger-list-missing-notes

Conversation

@SarveshaKumarKS

Copy link
Copy Markdown

Summary

  • return one None score per applicable output when attempt.notes["triggers"] is absent
  • add direct regression coverage for multiple outputs
  • update PIILeakExact's inherited-behaviour test to expect aligned unscored results

Root cause and impact

TriggerListDetector.detect initialised an empty result list, but its entire scoring loop was guarded by if "triggers" in attempt.notes. When a probe did not populate that note, the detector returned [] regardless of the number of outputs.

The evaluator enumerates detector scores, so an empty result list increments neither pass, fail, nor None counters. Returning [None] * len(all_outputs) preserves the detector/output alignment and records each output as unscorable. This change only handles an absent note; it does not alter the existing semantics of an explicitly empty trigger list.

Verification

  • Red proof on main: a two-output attempt with no triggers note returned []
  • Green proof after the fix: the same attempt returns [None, None]
  • python -m pytest tests/detectors/test_detectors_base.py tests/detectors/test_detectors_propile.py tests/detectors/test_detectors_trigger_list.py -q — 42 passed
  • python -m pytest tests/probes/test_probes_propile.py -q — 18 passed
  • python -m black --check -W 1 garak/detectors/base.py tests/detectors/test_detectors_propile.py tests/detectors/test_detectors_trigger_list.py — passed with Black 26.1.0
  • git diff --check — passed
  • Full tests/detectors run was environment-limited: 707 passed, 34 skipped, and 46 tests requiring external Hugging Face models or package-registry data failed because those resources were unavailable in the restricted test environment

Not a duplicate

AI assistance

OpenAI Codex assisted with investigation, implementation, testing, and drafting this description. The human submitter reviewed every changed line, understands and approves the change, and the commit includes both AI attribution and the required DCO sign-off.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Sarvesha Kumar Kombaiah Seetha <202573753+SarveshaKumarKS@users.noreply.github.com>
@SarveshaKumarKS
SarveshaKumarKS marked this pull request as ready for review July 30, 2026 00:14
@jmartin-tech jmartin-tech added the bug Something isn't working label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants