[benchmarking] Multiple fixes to stabilize the nightly benchmark suite

[rlratzel]

Summary

Bundles eight independent fixes uncovered while triaging the Curator
nightly benchmark suite against main (reference nemo-ci pipelines
52840568,
53098506,
53323567,
and 53333039).

Pass-rate progression: 23/37 → 28/37 → 6/8 (scoped) → 1/2 (scoped) → expected
2/2 with the latest commit.

1. Make undefined env vars in config non-fatal by default (b737202d)

• resolve_env_vars previously raised ValueError on the first undefined
  ${VAR} reference, halting the entire benchmark session even when the
  missing var was used by only a few entries.
• Default behavior is now to substitute an empty string and log a warning
  so unrelated entries can still run.
• Adds --strict-config-check CLI flag to run.py to restore the old
  fail-fast behavior.

2. New --entries-exact flag, exact-match entry filtering (ab1d4385, e8c6a52f, 7b351693)

Fixes a substring-aliasing bug in CI per-job invocations:

• --entries uses pytest's -k substring expression evaluator, so
  --entries audio_tagging_tts_xenna also matches
  audio_tagging_tts_xenna_repeat. In CI, where each per-job script runs
  --entries <entry-name>, this caused the non-_repeat job to also
  execute the _repeat entry, polluting that entry's per-entry results
  dir and crashing the subsequent legitimate _repeat SLURM job with
  Capture file ... already exists at Ray cluster setup.
• Adds --entries-exact accepting a comma-separated list of exact entry
  names. Every supplied name must match a configured (enabled) entry,
  otherwise the run aborts with a ValueError listing the unknown names
  alongside the available entry names.
• Mutually exclusive with --entries (CLI and Session.from_dict both
  enforce this).
• benchmarking/tools/ci_benchmark_launcher.sh switched from --entries
  to --entries-exact.
• Interactive --entries substring expression semantics are unchanged.

3. Bump too-tight timeout_s values for several entries

Multiple nightly-benchmark.yaml entries had timeout_s values that
didn't cover the actual wall time on the EOS H100 runner. These split
into two categories:

• Pre-existing tight limits that overshot during Ray teardown (Add style check #3.1).
• "No-op" timeouts — set back when the entry effectively did no work
  (e.g. rpv2 data was unreadable, or fuzzy_id_generator artifacts were
  stale and the test failed fast); once the underlying issue was fixed
  and the test ran for real, the historical ceiling was too tight (Add style check #3.2,
  Add style check #3.3).

3.1 ndd_ray_serve_dp4: 700 → 1200 (ab1d4385)

• SLURM-killed at the 700s --time ceiling ~13s after its benchmark
  subprocess succeeded — Ray teardown overhead pushed total wall over the
  wire.
• 1200s gives ~70% headroom over the observed warm-run wall.
• Reference failed job:
  https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/jobs/327777542

3.2 exact_dedup_identification: 500 → 1500 (21647b4f)

• The 500s value was set when the test was effectively a no-op (the rpv2
  dataset was unreadable due to filesystem permissions, so the test failed
  fast on stat()). Once rpv2 access was restored, the test SLURM-killed
  at 68% (516/755) of the "Inserting into shuffler" phase.
• Shuffler runs at ~1.83 it/s × 755 items = ~410s on EOS H100, plus Ray
  cluster setup + dataset stat + post-shuffle dedup compute + cleanup ≈
  realistic wall 800-1000s.
• 1500s gives ~50% headroom.
• Reference failed job:
  https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/jobs/329434352

3.3 dedup_removal_*: 1100/1500 → 1800/1800 (c2f3a6b0)

Once the upstream fuzzy_id_generator artifacts were refreshed (see
section 4 — the delete_scratch: false fix made this possible), the
dedup_removal tests actually ran the workload for the first time:

• dedup_removal_raydata: observed wall ~1419s (92% complete when killed
  at the prior 1100s ceiling) → 1800s (~27% headroom over the estimated
  1500s full wall).
• dedup_removal_xenna: passed at 1523s wall with the prior 1500s ceiling
  (zero margin) → 1800s as a preemptive bump to match raydata and absorb
  run-to-run variance.

Reference: nemo-ci pipeline 53333039 / leaf 53333328 — raydata killed by
SLURM TIME LIMIT during normal pipeline execution while xenna finished
with zero headroom.

4. Install lynx in the CI benchmark launcher (d0eac901)

The math benchmarks (math_preprocess, math_preprocess_classifier,
math_preprocess_llm_cleanup) shell out to the lynx text browser via
nemo_curator/stages/math/download/html_extractors/lynx.py for HTML
extraction. lynx is not in the Curator container, so those benchmarks
fail with RuntimeError: lynx executable not found in PATH.

lynx is GPL-licensed, so we deliberately do not bake it into the
redistributable Curator image. Instead it is installed transiently in the
existing benchmark container at CI run time, used during the run, and
discarded with the container. The published image stays GPL-free; the
apt-installed lynx only lives for the lifetime of each CI container.

5. (Reverted) Preserve scratch dir for fuzzy_dedup_identification (63691509 then a3c38ac6)

Initial commit 63691509 added delete_scratch: false to the
fuzzy_dedup_identification entry to keep its scratch artifacts around for
the downstream dedup_removal_* benchmarks. In practice the consumption
path is the canonical dataset path:

{datasets_path}/cleaned_exact_dedup_all_cc_fuzzy_output_nightly_container_paths/
  fuzzy_id_generator.json
  FuzzyDuplicateIds/

Operators promote a known-good fuzzy_dedup output to this path once, and
the dedup_removal entries consume it from there on every subsequent
pipeline — there is no same-pipeline dependency. Given that workflow, the
per-entry delete_scratch: false override is unnecessary and just leaves
unused data on lustre across runs. Reverted by a3c38ac6; the entry now
uses the session-level default (delete_scratch: true).

6. Range domain_label_games_count metric +/-5% (0c375b7e)

Both domain_classification_raydata and domain_classification_xenna
had a requirement on domain_label_games_count with exact_value: 149816.
Run-to-run output of the classifier drifts by several hundred to a few
thousand classifications, causing repeated false failures while the actual
benchmark (throughput, total docs, number of domains predicted) is
healthy.

Loosens the metric to a +/-5% range (min_value: 142325,
max_value: 157307) which still catches genuine regressions of the
classifier output while tolerating normal run-to-run variability.
domain_label_news_count is left at exact_value: 2817 pending further
investigation.

Verification

Across multiple nemo-ci pipelines:

• 53098506
  — first verification of commits 1, 2, 3.1, 3.2 (and Make NeMo-Curator installable in non GPU environments #4 below): 28/37
  pass (up from 23/37 baseline). Confirmed
  audio_tagging_tts_xenna_repeat works (--entries-exact), ndd_ray_serve_dp4
  works (timeout bump), exact_dedup_identification works (rpv2 + timeout),
  audio_tagging_tts_xenna works without HF_SECRET_KEY set.
• 53323567
  — scoped 8-entry run of commits 4, 5, 6: 6/8 pass (math_preprocess*
  all pass thanks to lynx install; domain_classification_* pass with the
  metric range; fuzzy_dedup_identification passes). The two failures
  (dedup_removal_*) were caused by a same-pipeline race condition with
  fuzzy_dedup_identification — fixed by operators manually copying the
  fresh fuzzy_id_generator.json artifacts to the canonical dataset path
  between runs.
• 53333039
  — scoped 2-entry run with fresh artifacts in place: dedup_removal_xenna
  passes (1523s, no margin); dedup_removal_raydata SLURM-killed by TIME
  LIMIT at 92% completion → addressed by commit c2f3a6b0 (Add style check #3.3).

A fresh pipeline against the latest tip (which includes c2f3a6b0) is
required to confirm both dedup_removal entries pass cleanly.

Test plan

Undefined env vars

• audio_tagging_tts_xenna ran successfully in 53098506 with
  HF_SECRET_KEY unset.
• Run with --strict-config-check; confirm it still exits with the
  original ValueError.
• Run an unmodified config (all env vars defined); confirm no
  behavior change.

--entries-exact

• CI per-job invocation ran exactly one entry per job in 53098506
  and 53323567; no cross-entry pollution observed.
• --entries-exact <typo> exits with an error listing the unknown
  name and the available entry names.
• --entries-exact a,b,c runs only those three entries in YAML order.
• Passing both --entries and --entries-exact exits with a
  "mutually exclusive" error.

Timeout bumps

• ndd_ray_serve_dp4 finished without TIME LIMIT cancellation in
  53098506.
• exact_dedup_identification finished without TIME LIMIT in 53098506.
• After c2f3a6b0 lands or in a follow-up run, confirm
  dedup_removal_raydata and dedup_removal_xenna both finish without
  TIME LIMIT cancellation. (Both passed in 53398187.)

lynx install

• math_preprocess* entries all passed in 53323567 — confirmed
  RuntimeError: lynx executable not found no longer fires.

fuzzy_dedup scratch retention (now reverted)

• One-time artifact promotion completed: fresh fuzzy_id_generator.json +
  FuzzyDuplicateIds/ written to the canonical dataset path
  ({datasets_path}/cleaned_exact_dedup_all_cc_fuzzy_output_nightly_container_paths/).
• dedup_removal_* consume the promoted artifacts in a clean
  pipeline (verified in 53398187).
• After artifact promotion, the per-entry delete_scratch: false
  override on fuzzy_dedup_identification is no longer needed and has
  been reverted (commit a3c38ac6).

domain_label_games_count range

• domain_classification_* both passed in 53323567 with the new
  range satisfied.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[praateekmahajan]

I'm wondering why this happened now?

[praateekmahajan]

Because even dynamo should not take 2700s rn since we upgraded versions to 1.1.0 (i.e.we can reduce that)

[rlratzel]

I need to investigate that. I know we had success on the DGX-A100 machine with 700s, but I don't know yet if a larger timeout is needed for the other machine running nightlies because it's slower in general, or if something else is causing a longer runtime on the other machine.