Skip to content

Allow callers to pass tokenizer_options in Text2TextGenerationPipeline #1675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Suh0161 wants to merge 1 commit into
base: main
Choose a base branch
from
+27 −2
Open

Allow callers to pass tokenizer_options in Text2TextGenerationPipeline #1675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions packages/transformers/src/pipelines/text2text-generation.js
View file
Open in desktop

@nico-martin nico-martin Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The public callback type still describes options only as generation parameters:

@param {Partial<import('../generation/parameters.js').GenerationFunctionParameters>} [options]

If tokenizer_options remains part of the pipeline API, it should get its own typedef, similar to how TextGenerationPipeline defines text-generation-specific params.

Original file line number Diff line number Diff line change
Expand Up @@ -72,24 +72,31 @@ export class Text2TextGenerationPipeline
}

const tokenizer = this.tokenizer;

// Callers may pass tokenizer_options as a top-level key inside generate_kwargs to
// control tokenization (e.g. max_length, truncation side) without touching generation

@nico-martin nico-martin Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says callers can control truncation side, and the PR description also mentions truncation_side. That key would be passed inside tokenizer_options, but the tokenizer implementation does not appear to use it. PreTrainedTokenizer._call() only reads truncation and max_length, and truncateHelper(...) always truncates from the right by shortening the array.

So tokenizer_options: { truncation_side: 'left' } would be silently ignored.

// parameters. We extract it here so it never leaks into model.generate().
const { tokenizer_options: caller_tokenizer_options, ...rest_generate_kwargs } = generate_kwargs;
const tokenizer_options = {
padding: true,
truncation: true,
...caller_tokenizer_options,
};

let inputs;
if (this.task === 'translation' && '_build_translation_inputs' in tokenizer) {
// TODO: move to Translation pipeline?
// Currently put here to avoid code duplication
// @ts-ignore
inputs = tokenizer._build_translation_inputs(texts, tokenizer_options, generate_kwargs);
inputs = tokenizer._build_translation_inputs(texts, tokenizer_options, rest_generate_kwargs);
} else {
inputs = tokenizer(texts, tokenizer_options);
}

const outputTokenIds = await this.model.generate({
...inputs,
...this._default_generation_config,
...generate_kwargs,
...rest_generate_kwargs,
});
return tokenizer
.batch_decode(/** @type {Tensor} */ (outputTokenIds), {
Expand Down
18 changes: 18 additions & 0 deletions packages/transformers/tests/pipelines/test_pipelines_text2text_generation.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,24 @@ export default () => {
},
MAX_TEST_EXECUTION_TIME,
);

it(
"tokenizer_options are forwarded to the tokenizer",
async () => {
const text = "This is a test.";
// max_length=3 forces aggressive truncation via tokenizer_options;
// the model should still run without error and return a string result.
// tokenizer_options is extracted before generate() is called so it
// never leaks into GenerationFunctionParameters.
const output = await pipe(text, {

@nico-martin nico-martin Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test would still pass if tokenizer_options were ignored, because it only checks that generation returns an array with a string. To protect this feature, please assert a visible effect of the tokenizer option, or spy/mock the tokenizer call and verify it receives { truncation: true, max_length: 3 } while model.generate() does not receive tokenizer_options.

max_new_tokens: 5,
tokenizer_options: { truncation: true, max_length: 3 },
});
expect(Array.isArray(output)).toBe(true);
expect(typeof output[0].generated_text).toBe("string");
},
MAX_TEST_EXECUTION_TIME,
);
});

afterAll(async () => {
Expand Down