tokenizers.js ignores byte_fallback for Unigram models

[sodiumplus3]

Feature request

tokenizers.js appears to ignore byte_fallback for Unigram models.

Some SentencePiece / Unigram tokenizers include the following configuration in tokenizer.json:

{
  "type": "Unigram",
  "byte_fallback": true
}

However, the current JavaScript implementation seems to fall back to a single <unk> token for out-of-vocabulary characters, instead of expanding them into UTF-8 byte tokens such as <0xXX>.

For example, if 🍣 is not in the vocab, I would expect it to be encoded as:

<0xF0> <0x9F> <0x8D> <0xA3>

rather than as a single token.

It looks like byte_fallback is not initialized or used for Unigram models in the same way as it is for BPE models, and OOV handling in the Unigram lattice path currently falls back to <unk>.

Would you consider supporting byte_fallback for Unigram models as a compatibility feature with SentencePiece / HuggingFace tokenizers?

Motivation

This matters because some exported tokenizer configs explicitly set byte_fallback=true.

For those models, tokenizers.js can currently produce different tokenization results from the Python / Rust implementations when the input contains OOV characters. This can lead to different token IDs being passed to the model for the same input text.

The intended use case is to make tokenizers.js compatible with Unigram tokenizer files that rely on byte fallback behavior, especially for multilingual or SentencePiece-based models where unknown Unicode characters should be represented as byte tokens instead of <unk>.

Your contribution

I have a draft implementation that adds byte fallback handling for Unigram models.

Since this changes the tokenization output for OOV characters when byte_fallback=true, I wanted to confirm the intended direction before opening a PR. If this behavior is considered desirable, I would be happy to submit a PR with the implementation and tests.

[xenova]

Hi there 👋 thanks for the report. Can you give an example of a model id (or tokenizer.json) which causes this issue?

[sodiumplus3]

@xenova
Hi, thanks for the quick response!
I found this issue while using the cl-nagoya/ruri-v3-30m model.

With Python transformers, the tokenizer expands the OOV emoji into UTF-8 byte fallback tokens as expected.

import transformers
import tokenizers

print(transformers.__version__)
print(tokenizers.__version__)

from transformers import AutoTokenizer

model_id = "cl-nagoya/ruri-v3-30m"
text = "寿司🍣が好き"  # Means "I like sushi."

tokenizer = AutoTokenizer.from_pretrained(model_id)

encoded = tokenizer(text, add_special_tokens=False)

print("text:", text)
print("input_ids:", encoded["input_ids"])
print("tokens:", tokenizer.convert_ids_to_tokens(encoded["input_ids"]))

Output:

4.57.6
0.22.2
text: 寿司🍣が好き
input_ids: [13038, 255, 174, 156, 178, 13350]
tokens: ['寿司', '<0xF0>', '<0x9F>', '<0x8D>', '<0xA3>', 'が好き']

However, with Transformers.js, the same input is tokenized differently. The emoji remains as a single token, and convert_tokens_to_ids returns undefined for it.

// sample.mjs
import { AutoTokenizer } from "@huggingface/transformers";

const modelId = "cl-nagoya/ruri-v3-30m";
const text = "寿司🍣が好き";

const tokenizer = await AutoTokenizer.from_pretrained(modelId);

const tokens = tokenizer.tokenize(text);
const ids = tokenizer.convert_tokens_to_ids(tokens);

console.log("text:", text);
console.log("tokens:", tokens);
console.log("input_ids:", ids);

Output:

text: 寿司🍣が好き
tokens: [ '寿司', '🍣', 'が好き' ]
input_ids: [ 13038, undefined, 13350 ]

Environment:

Node.js: v22.22.3
Package manager: pnpm
@huggingface/transformers: 4.2.0

This looks like byte_fallback=true is applied by Python/Rust tokenizers for this Unigram tokenizer, but not by Transformers.js.