fix(detectors): preserve TriggerListDetector result alignment - #2000
Open
SarveshaKumarKS wants to merge 1 commit into
Open
fix(detectors): preserve TriggerListDetector result alignment#2000SarveshaKumarKS wants to merge 1 commit into
SarveshaKumarKS wants to merge 1 commit into
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Sarvesha Kumar Kombaiah Seetha <202573753+SarveshaKumarKS@users.noreply.github.com>
SarveshaKumarKS
marked this pull request as ready for review
July 30, 2026 00:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nonescore per applicable output whenattempt.notes["triggers"]is absentPIILeakExact's inherited-behaviour test to expect aligned unscored resultsRoot cause and impact
TriggerListDetector.detectinitialised an empty result list, but its entire scoring loop was guarded byif "triggers" in attempt.notes. When a probe did not populate that note, the detector returned[]regardless of the number of outputs.The evaluator enumerates detector scores, so an empty result list increments neither pass, fail, nor
Nonecounters. Returning[None] * len(all_outputs)preserves the detector/output alignment and records each output as unscorable. This change only handles an absent note; it does not alter the existing semantics of an explicitly empty trigger list.Verification
main: a two-output attempt with notriggersnote returned[][None, None]python -m pytest tests/detectors/test_detectors_base.py tests/detectors/test_detectors_propile.py tests/detectors/test_detectors_trigger_list.py -q— 42 passedpython -m pytest tests/probes/test_probes_propile.py -q— 18 passedpython -m black --check -W 1 garak/detectors/base.py tests/detectors/test_detectors_propile.py tests/detectors/test_detectors_trigger_list.py— passed with Black 26.1.0git diff --check— passedtests/detectorsrun was environment-limited: 707 passed, 34 skipped, and 46 tests requiring external Hugging Face models or package-registry data failed because those resources were unavailable in the restricted test environmentNot a duplicate
notes["triggers"]path inTriggerListDetector.garak/detectors/base.py.Nonescores in evaluator metrics, not the detector's missing result entries.attempt.notes["triggers"]#1147 standardised consumers onnotes["triggers"]but did not add this guard toTriggerListDetector.MarkdownExfilContentandnotes["terms"].AI assistance
OpenAI Codex assisted with investigation, implementation, testing, and drafting this description. The human submitter reviewed every changed line, understands and approves the change, and the commit includes both AI attribution and the required DCO sign-off.