Allow callers to pass tokenizer_options in Text2TextGenerationPipeline

[Suh0161]

Closes #1096

Feature: tokenizer_options passthrough in Text2TextGenerationPipeline

The tokenizer call inside _call() previously used hardcoded options with no way to override them. Callers can now pass tokenizer_options alongside generation kwargs to control things like max_length and truncation_side.

tokenizer_options is extracted from generate_kwargs before model.generate() is called so it never leaks into GenerationFunctionParameters. Pipeline defaults (padding: true, truncation: true) still apply and caller values take precedence.

const generator = await pipeline('text2text-generation', 'Xenova/LaMini-Flan-T5-783M');
const output = await generator('Translate to French: Hello world', {
  max_new_tokens: 100,
  tokenizer_options: { max_length: 64, truncation: true },
});

[nico-martin]

HI @Suh0161, thanks for looking into this!

I actually think the change is useful. But there is also an API-design question: if we want to stay close to Python Transformers, explicit preprocess options like truncation are a better fit than a broad tokenizer_options bag.

@xenova, what do you think?

[nico-martin]

The comment says callers can control truncation side, and the PR description also mentions truncation_side. That key would be passed inside tokenizer_options, but the tokenizer implementation does not appear to use it. PreTrainedTokenizer._call() only reads truncation and max_length, and truncateHelper(...) always truncates from the right by shortening the array.

So tokenizer_options: { truncation_side: 'left' } would be silently ignored.

[nico-martin]

The public callback type still describes options only as generation parameters:

@param {Partial<import('../generation/parameters.js').GenerationFunctionParameters>} [options]

If tokenizer_options remains part of the pipeline API, it should get its own typedef, similar to how TextGenerationPipeline defines text-generation-specific params.

[nico-martin]

This test would still pass if tokenizer_options were ignored, because it only checks that generation returns an array with a string. To protect this feature, please assert a visible effect of the tokenizer option, or spy/mock the tokenizer call and verify it receives { truncation: true, max_length: 3 } while model.generate() does not receive tokenizer_options.