feat(probes): add code-switching jailbreak probe (codeswitch)

[Yugesh-reddy]

Status

Draft — awaiting design buy-in on #1994.

Issue #1994 and this PR were opened in the same session; there has been no maintainer reply on #1994 yet. Per contributing guidance, please treat the issue as the proposal thread. I will only mark this PR ready after scope / data-hosting guidance.

Summary

• Adds probes.codeswitch with CodeSwitchFull (inactive) and CodeSwitch (active, pruned to soft_probe_prompt_cap).
• Tests whether refusal training survives intra-sentential language mixing (code-switching within a sentence).
• Distinct from garak langprovider whole-prompt translation: prompts are pre-authored and used verbatim.
• Docs + unit tests included. doc_uri → https://github.com/Yugesh-reddy/MediCS-Red-Teaming (no published paper yet).

Prompt count / split (exact)

MediCS-500 is the project name, not the number of prompts in this PR.

What	Count
Prompts currently shipped in garak/data/codeswitch/codeswitch_multi.txt	2,665
Source	attack_prompt field from medics_500_full.jsonl (full attack corpus)
MediCS held-out eval split (not what is shipped today)	533

So this draft currently includes the full corpus, not only the held-out eval split. That means some prompts may overlap MediCS training/defense construction. If maintainers prefer a contamination-safe probe, I can switch the shipped file to the 533 held-out prompts only (and keep Full/pruned naming aligned to that).

lang convention

Open question on #1994 is still unanswered by maintainers. In this draft I set:

lang = "*"

Why: these prompts inherently mix languages and must not be run through langprovider translation. With lang = "en", a non-English run.target_lang would translate them and destroy the attack surface. lang = "*" matches other verbatim/multilingual probes and falls back to the passthrough language provider. Happy to change if maintainers prefer lang = "en" (carrier) plus another no-translate mechanism.

Not a duplicate

Checked open issues/PRs for codeswitch / MediCS / code-switching. No matching probe PR. Closest open work (sandwich multilingual jailbreak, Korean jailbreak probes) uses different techniques.

Provenance

Prompts from https://github.com/Yugesh-reddy/MediCS-Red-Teaming (author-contributed; see garak/data/codeswitch/README.md). Harmful medical content — hosting policy still open on #1994.

AI assistance

AI assistance was used while preparing this contribution. A human author reviewed every changed line and ran the tests below.

Test plan

• python -m pytest tests/probes/test_probes_codeswitch.py -v (5 passed, including lang == "*" guard)
• Docs/plugin docstring checks for codeswitch (passed)
• Probe appears in garak --list_probes (codeswitch.CodeSwitch, codeswitch.CodeSwitchFull)
• black --config pyproject.toml on probe/test Python files
• Maintainer guidance on probe: intra-sentential code-switching jailbreak #1994 (scope + data hosting + preferred split)
• Maintainer review of medical-domain harmful prompt content and detector choice (mitigation.Prefixes)

[Copilot]

Pull request overview

Adds a new codeswitch probe module to evaluate jailbreak/refusal robustness under intra-sentential multilingual code-switching, backed by a new MediCS-derived prompt dataset, with docs and basic unit tests to validate loading and pruning behaviour.

Changes:

• Introduces garak.probes.codeswitch with CodeSwitchFull (inactive) and CodeSwitch (active, pruned to soft_probe_prompt_cap).
• Adds MediCS-500 code-switched prompts under garak/data/codeswitch/ with provenance notes.
• Registers Sphinx docs for the new probe and adds probe-focused unit tests.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
garak/probes/codeswitch.py	New probe implementation that loads/verbatim-serves code-switched prompts and provides a pruned active variant.
garak/data/codeswitch/README.md	Documents prompt provenance and “verbatim/no-translation” handling expectations.
docs/source/probes/codeswitch.rst	Adds Sphinx automodule page for the new probe.
docs/source/index_probes.rst	Includes codeswitch in the probes documentation index.
tests/probes/test_probes_codeswitch.py	Adds unit tests covering prompt loading, pruning, and skipping comments/blank lines.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Yugesh-reddy]

Addressed the review note on lang: set to `*` so prompts stay verbatim and are not run through `langprovider` translation (same pattern as other multilingual/verbatim probes). Added a unit test guarding this.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

tests/probes/test_probes_codeswitch.py:16

• Most probe tests include terse assertion messages to make failures easier to diagnose (e.g. tests/probes/test_probes_smuggling.py:15-18). These asserts currently have no message, which makes CI failures harder to interpret.

def test_codeswitch_pruned_probes_have_fewer_prompts():
    p_full = garak._plugins.load_plugin("probes.codeswitch.CodeSwitchFull")
    p_pruned = garak._plugins.load_plugin("probes.codeswitch.CodeSwitch")
    assert len(p_pruned.prompts) <= len(p_full.prompts)

tests/probes/test_probes_codeswitch.py:21

• Most probe tests include terse assertion messages to make failures easier to diagnose (e.g. tests/probes/test_probes_smuggling.py:10). These asserts currently have no message, which makes CI failures harder to interpret.

def test_codeswitch_pruned_prompt_count_within_cap():
    p = garak._plugins.load_plugin("probes.codeswitch.CodeSwitch")
    assert len(p.prompts) <= p.soft_probe_prompt_cap

tests/probes/test_probes_codeswitch.py:27

• Most probe tests include terse assertion messages to make failures easier to diagnose (e.g. tests/probes/test_probes_smuggling.py:10). These asserts currently have no message, which makes CI failures harder to interpret.

def test_codeswitch_loader_skips_blank_and_comment_lines():
    p = garak._plugins.load_plugin("probes.codeswitch.CodeSwitchFull")
    assert all(prompt.strip() for prompt in p.prompts)
    assert not any(prompt.startswith("#") for prompt in p.prompts)

garak/probes/codeswitch.py:3

• New probe modules in this repo commonly include SPDX copyright/license header lines at the top of the file (for example garak/probes/dra.py:1-2). garak/probes/codeswitch.py currently starts directly with the module docstring, which makes licensing metadata inconsistent across probes.

"""**Code-switching**

These attacks mix two or more languages within a single instruction to test whether

tests/probes/test_probes_codeswitch.py:10

• Most probe tests include terse assertion messages to make failures easier to diagnose (e.g. tests/probes/test_probes_smuggling.py:10). These asserts currently have no message, which makes CI failures harder to interpret.

This issue also appears in the following locations of the same file:

• line 12
• line 18
• line 23

def test_codeswitch_full_loads_prompts():
    p_full = garak._plugins.load_plugin("probes.codeswitch.CodeSwitchFull")
    assert len(p_full.prompts) > 0

[Yugesh-reddy]

Converting this PR to draft.

I opened it too early relative to issue #1994. Per contributing guidance, I'd like design buy-in on scope and especially data hosting for the ~2.6k harmful medical prompts before asking for a merge review.

Please treat #1994 as the proposal thread; happy to mark this ready again (or close/reopen) once maintainers confirm direction.