fix(ci): gate fastokens benches behind RUN_BENCH

[renflowerz]

Overview:

Gate the two fastokens benches in tokenizer_simple behind RUN_BENCH=1 so cargo test --all-targets in CI no longer depends on a Hugging Face download.

Details:

cargo test --all-targets executes tokenizer_simple, whose two fastokens benches download Qwen/Qwen3-0.6B from Hugging Face at runtime. A transient Hugging Face Xet/CAS 403 caused the same failure on both main (run 29290753192) and an unrelated PR (run 29291246536).

Per review discussion, the fix is to gate the network-dependent benches instead of caching the tokenizer in CI:

• fastokens_encode and fastokens_batch_encode now skip unless RUN_BENCH=1 is set, matching the existing opt-in pattern in tokenizer_dataset.rs
• the other five benches in the file use in-tree tokenizers and keep running under cargo test --all-targets
• no workflow changes

Why not an in-tree tokenizer: the fastokens parity check needs a byte-level BPE tokenizer (ByteLevel pre-tokenization, GPT/Qwen style). TinyLlama (the only real tokenizer in tests/data) is BPE but sentencepiece-style (metaspace + byte-fallback, no pre-tokenizer), which fastokens does not handle, and the byte-level mocks have empty vocabs. If the fastokens-vs-HF parity assertion is wanted back in CI later, a small trained byte-level BPE mock in tests/data would be the way (possible follow-up).

Validation:

• RUN_BENCH unset: both fastokens benches print a skip message and return, no network access
• RUN_BENCH=1: behavior unchanged from before
• rustfmt clean
• signed-off, GPG-signed single commit

Where should the reviewer start?

lib/llm/benches/tokenizer_simple.rs: the RUN_BENCH guards at the top of fastokens_encode and fastokens_batch_encode.

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi renflowerz! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

The pre-merge workflow caches, validates, downloads, and passes a Qwen tokenizer to root-directory Rust tests through TOKENIZER_PATH.

Changes

Qwen tokenizer CI preparation

Layer / File(s)	Summary
Prepare and use the tokenizer .github/workflows/pre-merge.yml	The root matrix job restores and verifies tokenizer.json, downloads it from Hugging Face when needed, exports its path, and sets TOKENIZER_PATH before running Cargo tests.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title mentions CI and RUN_BENCH gating, but the changes actually add tokenizer caching in the workflow, so it's misleading.	Rename it to reflect the workflow/tokenizer caching change, e.g. "fix(ci): cache Qwen tokenizer for rust tests".

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description includes the required sections and covers overview, details, reviewer start, and related issues.

---

_{Comment @coderabbitai help to get the list of available commands.}

[renflowerz]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/pre-merge.yml:
- Line 190: Update the tokenizer download curl command in the pre-merge workflow
to include explicit connection and overall transfer timeouts, while preserving
its existing retry, URL, and output-file behavior.
- Around line 166-171: Update the “Cache Qwen tokenizer” step to use a
repair-capable cache flow: avoid restoring the corrupted exact-key entry, and
save the successfully redownloaded tokenizer under a distinct repair-specific
key or otherwise ensure the healed file replaces the bad cache. Preserve the
existing path and root-directory condition while making subsequent runs restore
the repaired tokenizer.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bd9f32d4-eabf-4bdf-9bf1-bf9b4fafa45f

📥 Commits

Reviewing files that changed from the base of the PR and between 60586fa and 5740458.

📒 Files selected for processing (1)

• .github/workflows/pre-merge.yml

[renflowerz]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.