Summary
On translated runs, a language-specific detector can score a reverse-translated output while the corresponding hitlog entry records only the raw model output.
The generation index remains aligned (including after #1959), so this is not a wrong-generation attribution bug. The auditability problem is that the hitlog does not contain the output representation selected for detection. A hit can therefore be difficult to explain or reproduce from the hitlog alone.
Current behavior
Language-aware detectors commonly select inputs through Attempt.outputs_for(self.lang_spec). For example, StringDetector.detect() uses:
all_outputs = attempt.outputs_for(self.lang_spec)
When the requested detector language differs from the prompt/model language, outputs_for() returns reverse_translation_outputs.
The evaluator later enumerates the resulting scores but writes:
"output": asdict(attempt.outputs[idx]),
Consequently:
score may have been calculated from reverse_translation_outputs[idx];output contains attempt.outputs[idx], the raw model generation;- the hitlog does not identify or include the representation used for detection.
Relevant code:
Basic example
Construct an attempt whose model-facing language differs from an English-only detector:
attempt = Attempt(prompt=Message("prompt", lang="es"))
attempt.outputs = [Message("respuesta sin la palabra inglesa", lang="es")]
attempt.reverse_translation_outputs = [
Message("reverse translation containing TRIGGER", lang="en")
]
detector = StringDetector(["TRIGGER"])
detector.lang_spec = "en"
scores = detector.detect(attempt)The detector scores the English reverse translation and produces a hit. After assigning the score and evaluating the attempt, the hitlog's output field contains the Spanish model output. It does not contain the English representation where TRIGGER matched, nor metadata identifying the representation used for detection.
The full report serializes reverse_translation_outputs, so the evidence can sometimes be reconstructed by joining records using the attempt ID and index. The standalone hitlog record is not self-explanatory.
Motivation
Keep output as the actual model generation, but make a translated-run hit independently auditable by recording the detector-scoped output selected before detector-specific preprocessing.
One possible backward-compatible shape would be an optional field such as:
{
"output": {"text": "raw model generation", "lang": "es"},
"detector_input": {"text": "reverse-translated representation", "lang": "en"},
"detector_input_source": "reverse_translation_outputs"
}The field names and storage location are open for discussion.
Design consideration
The evaluator currently receives only detector_name and scores; it does not receive the detector instance or its lang_spec. Correctly identifying the selected representation may therefore require carrying detector-input provenance through the harness/attempt data rather than having the evaluator infer it after the fact.
I would appreciate maintainer guidance on the preferred representation before preparing a patch.
Dedupe notes
Summary
On translated runs, a language-specific detector can score a reverse-translated output while the corresponding hitlog entry records only the raw model output.
The generation index remains aligned (including after #1959), so this is not a wrong-generation attribution bug. The auditability problem is that the hitlog does not contain the output representation selected for detection. A hit can therefore be difficult to explain or reproduce from the hitlog alone.
Current behavior
Language-aware detectors commonly select inputs through
Attempt.outputs_for(self.lang_spec). For example,StringDetector.detect()uses:When the requested detector language differs from the prompt/model language,
outputs_for()returnsreverse_translation_outputs.The evaluator later enumerates the resulting scores but writes:
Consequently:
scoremay have been calculated fromreverse_translation_outputs[idx];outputcontainsattempt.outputs[idx], the raw model generation;Relevant code:
Basic example
Construct an attempt whose model-facing language differs from an English-only detector:
The detector scores the English reverse translation and produces a hit. After assigning the score and evaluating the attempt, the hitlog's
outputfield contains the Spanish model output. It does not contain the English representation whereTRIGGERmatched, nor metadata identifying the representation used for detection.The full report serializes
reverse_translation_outputs, so the evidence can sometimes be reconstructed by joining records using the attempt ID and index. The standalone hitlog record is not self-explanatory.Motivation
Keep
outputas the actual model generation, but make a translated-run hit independently auditable by recording the detector-scoped output selected before detector-specific preprocessing.One possible backward-compatible shape would be an optional field such as:
{ "output": {"text": "raw model generation", "lang": "es"}, "detector_input": {"text": "reverse-translated representation", "lang": "en"}, "detector_input_source": "reverse_translation_outputs" }The field names and storage location are open for discussion.
Design consideration
The evaluator currently receives only
detector_nameand scores; it does not receive the detector instance or itslang_spec. Correctly identifying the selected representation may therefore require carrying detector-input provenance through the harness/attempt data rather than having the evaluator infer it after the fact.I would appreciate maintainer guidance on the preferred representation before preparing a patch.
Dedupe notes