Skip to content

feat: Add HuggingFace-style skip_special_tokens and batch_decode support (#403)#1186

Open
lakshitsoni26 wants to merge 2 commits into
mlfoundations:mainfrom
lakshitsoni26:feature/tokenizer-decode-improvements
Open

feat: Add HuggingFace-style skip_special_tokens and batch_decode support (#403)#1186
lakshitsoni26 wants to merge 2 commits into
mlfoundations:mainfrom
lakshitsoni26:feature/tokenizer-decode-improvements

Conversation

@lakshitsoni26

@lakshitsoni26 lakshitsoni26 commented Jun 28, 2026

Copy link
Copy Markdown
  • Fixes Improve tokenizer decode #403 by adding batch_decode, skip_special_tokens, and clean_up_tokenization_spaces to the tokenizer API.
  • Implemented a clean_up_tokenization helper to exactly mimic HuggingFace's internal string-cleaning logic (stripping spurious spaces before punctuation).
  • Added batch_decode and the new decoding arguments to SimpleTokenizer and TikTokenTokenizer.
  • Added pass-through wrappers for the new arguments in HFTokenizer and SigLipTokenizer.
  • Bug Fix: Fixed an edge case where padding tokens could accidentally bleed into the output string in SimpleTokenizer.
  • Testing: Added comprehensive unit tests in tests/test_tokenizer_decode.py covering all tokenizer variants (100% test coverage for the new features).
  • Zero Regressions: Preserves exact backwards compatibility by defaulting the new boolean arguments to False across all signatures.
Screenshot 2026-06-28 at 11 28 12 AM Screenshot 2026-06-28 at 11 29 22 AM Screenshot 2026-06-28 at 2 03 25 PM

@rwightman

Copy link
Copy Markdown
Collaborator

@lakshitsoni26 so there are some risks here, and decisions to be made, it's not a 'zero regression' case... there's also some shifting ground re other significant additions (e.g. #2713) so am going to defer tokenizer cleanup until a bit later

@lakshitsoni26

Copy link
Copy Markdown
Author

Understood! Thanks for taking a look. I'll leave this PR open for now — just ping me whenever the dust settles on #2713 and you are ready to revisit the tokenizer cleanup. Happy to update this PR to resolve any conflicts then!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve tokenizer decode

2 participants