NIM generator aborts entire probe on a single transient HTTP error (408), and reports a misleading error message

[eastonl-nv]

Summary

When the target endpoint returns a transient HTTP error (observed: 408 REQUEST TIMEOUT) for a
single generation, garak.generators.nim raises a fatal GarakException that propagates out of the
probe's worker pool and aborts the entire probe run mid-way. In addition, the exception message
("NIM generation failed. Is the model name spelled correctly?") is misleading — the model name was
correct; the real cause was an HTTP timeout.

Environment

• garak version path: /opt/venv/lib/python3.12/site-packages/garak/
• Target: an OpenAI-compatible endpoint fronted by a load-balancing proxy, backed by a slow reasoning model
• Probe: leakreplay.LiteratureCloze (256 samples), max_tokens=32768
• Run context: garak run driven by nemo-evaluator-launcher on Slurm

What happened

The LB proxy in front of the model returned 408 REQUEST TIMEOUT after the model (a slow reasoning
model at 32k max tokens) exceeded the request timeout. garak's NIM generator turned that single failed
generation into a fatal exception, killing the probe at attempt 48/256. No detector/eval pass ran,
so the run produced attempts but zero eval entries in results.report.jsonl.

Evidence

Backend error (client/proxy log):

httpx.HTTPStatusError: Client error '408 REQUEST TIMEOUT' for url 'http://localhost:3825/chat/completions'

garak traceback (artifacts/garak/garak.log):

2026-07-14 03:31:10  CRITICAL  NIM generation failed. Is the model name spelled correctly?
...
  File "/opt/venv/lib/python3.12/site-packages/garak/generators/nim.py", line 101, in _call_model
    raise GarakException(f"🛑 {msg}") from oe
garak.exception.GarakException: 🛑 NIM generation failed. Is the model name spelled correctly?

The above exception was the direct cause of the following exception:

  File "/opt/venv/lib/python3.12/site-packages/garak/cli.py", line 614, in main
    command.probewise_run(...)
  File "/opt/venv/lib/python3.12/site-packages/garak/harnesses/probewise.py", line 109, in run
  File "/opt/venv/lib/python3.12/site-packages/garak/harnesses/base.py", line 151, in run
    attempt_results = probe.probe(model)
  File "/opt/venv/lib/python3.12/site-packages/garak/probes/base.py", line 449, in probe
    attempts_completed = self._execute_all(attempts_todo)
  File "/opt/venv/lib/python3.12/site-packages/garak/probes/base.py", line 335, in _execute_all
    for result in attempt_pool.imap_unordered(...)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 873, in next
    raise value
garak.exception.GarakException: 🛑 NIM generation failed. Is the model name spelled correctly?

Resulting report (results.report.jsonl) entry-type counts — note no eval:

{'start_run setup': 1, 'init': 1, 'attempt': 48}

(4 occurrences of NIM generation failed appeared in garak.log over the run.)

Root cause (confirmed against main)

The retry gap is specific. OpenAICompatible._call_model in garak/generators/openai.py retries only a
fixed set of exceptions via its backoff decorator:

@backoff.on_exception(
    backoff.fibo,
    (
        openai.RateLimitError,        # 429
        openai.InternalServerError,   # 5xx
        openai.APITimeoutError,       # client-side read timeout (no HTTP status)
        openai.APIConnectionError,
        garak.exception.GeneratorBackoffTrigger,
    ),
    max_value=70,
)

A server-issued HTTP 408 REQUEST TIMEOUT does not map to any of these. In the OpenAI SDK, 408 has no
dedicated exception subclass, so it is raised as a generic openai.APIStatusError. That class is:

• not in the backoff tuple above → it is never retried, and
• not handled by any specific except block in OpenAICompatible._call_model (which only catches
  AuthenticationError/PermissionDeniedError, BadRequestError, and JSONDecodeError).

So the 408 propagates up into NVOpenAIChat._call_model (garak/generators/nim.py), where the catch-all
except Exception (see below) turns it into a fatal GarakException that aborts the whole probe.

garak/generators/nim.py, NVOpenAIChat._call_model (current main):

except Exception as oe:
    msg = "NIM generation failed. Is the model name spelled correctly?"
    logging.critical(msg, exc_info=oe)
    raise GarakException(f"🛑 {msg}") from oe

Impact

• A single transient endpoint error (408, or other transient APIStatusError codes such as 502/503/504
  that are not InternalServerError) discards an entire probe's partial work — expensive for slow probes
  that already ran for over an hour.
• The misleading "Is the model name spelled correctly?" message sends operators debugging a
  configuration problem that does not exist, when the real issue is a transient HTTP timeout.

Requests

1. Add transient HTTP status codes to the retry/backoff set. At minimum 408, and consider 502/503/504.
   These currently arrive as openai.APIStatusError and bypass the backoff decorator entirely. Retrying
   them (with the existing fibo backoff) would avoid killing the probe on a single transient hiccup.
2. Do not abort the whole probe on a single unhandled generation error. Rather than the NIM subclass's
   catch-all except Exception raising a fatal GarakException out of the worker pool, record the attempt
   as failed and continue (or retry), so one bad request does not discard 250+ completed samples.
3. Fix the error message. Report the actual HTTP status / underlying error (e.g. "HTTP 408 Request
   Timeout from ") instead of the generic "Is the model name spelled correctly?" — the latter is
   misleading for any non-2xx that is not a genuine 404/400.

Notes

This surfaced downstream as a "job COMPLETED but no results" anomaly because the Slurm wrapper reported
exit 0 despite this crash (that wrapper behavior is a separate issue filed against nemo-evaluator-launcher).
The garak-side ask here is independent: be resilient to transient endpoint errors and report the real cause.

Related but distinct: #1466 (a separate NIM-client crash — file-descriptor leak on long runs).

[jmartin-tech]

Some initial thoughts, there are definitely some improvements that are viable. Support for behaviors of all possible proxy configurations that a target may have in the stack is not in scope, however I can see broad value in general HTTP code support for this generator.

Item 1 is a reasonable request, the currently configured backoff handing should cover most 4xx and 5xx errors that are actually transient and possible to retry. Extended HTTP error code specific handling may be appropriate in OpenAICompatible when openai.APIStatusError is raised.

Item 2 while understandable is too broad to be guaranteed though is possible to improve towards. Item 1 will likely improve handling for this generator.

Item 3 may not be in scope based on this issue, or to be more specific item 1 would make it a moot point as the HTTP status would be handled higher in the stack.

[eastonl-nv]

Thank you @jmartin-tech, so can we improve for item 1?

[jmartin-tech]

Yes, a quick looks suggests that a refactor of OpenAICompatible to catch the openai.APIStatusError response and inspect the error code to better react and either backoff or return None indicating the generation failed is viable. Happy to review a PR if you want to take this on, otherwise I should be able to work on one in the next week or so.

[Varshith-Kali]

PR #1973 addresses items 1 and 3 from this issue:

1. Adds openai.APIStatusError to the backoff retry tuple with a giveup predicate that retries only transient HTTP status codes (408, 502, 503, 504). Non-transient codes (404, 401, etc.) give up immediately.

2. Improves the NIM generator catch-all error message to include the actual HTTP status code (e.g. "NIM generation failed (HTTP 408)") instead of the generic "Is the model name spelled correctly?" message.

Includes 5 unit tests for the giveup predicate logic. Happy to adjust the approach if you'd prefer a different shape.