Output Generation Guided by JSON Schema

[crisdesivo]

Feature request

I propose the addition of a feature to transformers.js that allows users to enforce a specified JSON schema during the text generation process. This would ensure that the model's output is syntactically correct JSON and conforms to the provided schema.

Motivation

Many applications require LLMs to produce predictable, valid JSON output for tasks like data extraction, API response generation, and populating structured data fields.

Your contribution

Afraid not.

[JackNeto]

Any progress on this?

[EmranAhmed]

Any update about this?

[nico-martin]

Thanks for the remimder. I hadn’t noticed this issue. But it looks like a useful adjustment. I’ll have a closer look at it over the next few days.

[nico-martin]

Hi everyone, I spent some time working on a first draft of this feature.

Branch: https://github.com/huggingface/transformers.js/tree/feat/response_format

The draft adds an OpenAI-style response_format option to the text-generation pipeline, backed by grammar-constrained decoding:

await pipe(messages, {
  max_new_tokens: 1024,
  response_format: {
    type: "json_schema",
    json_schema: schema,
  },
});

It also supports the schema-less JSON mode:

response_format: { type: "json_object" }

Under the hood, this uses llguidance for constrained decoding. I chose llguidance because it is the same grammar engine used for structured output in Chrome's Prompt API work, and it is designed for efficient per-token mask computation during generation.

To make this usable from Transformers.js, I also spent some time porting llguidance to WASM and publishing a JS package:

• https://github.com/nico-martin/llguidance-js
• https://www.npmjs.com/package/llguidance

The Transformers.js draft loads that package lazily when response_format is used. The branch also adds the small generation-loop hooks needed by llguidance:

• one hook after a token is sampled, so the grammar state can commit the token, and
• one hook to stop generation once the JSON grammar reaches a valid final state.

There is a working local test script here:

node test/response_format.js

It currently runs two examples with onnx-community/gemma-4-E2B-it-ONNX:

• A simple schema for an array of strings.
• A more complex schema for a book list, where each item has title, author, and summary fields.

This is definitely still an early draft rather than something I would consider production-ready. But it works! Some next steps I think would be useful:

• Test this in real browser environments, not only local Node/WebGPU runs.
• Evaluate llguidance bundle size, startup cost, tokenizer bridge cost, and per-token mask latency in Transformers.js.
• Add proper end-to-end tests with small models and representative schemas.
• Expand validation and fail-fast errors for unsupported JSON Schema features.
• Decide whether llguidance should be a required dependency, optional dependency, or bundled differently.
• Investigate whether a smaller llguidance-json build could support only JSON / JSON Schema constraints.
• Alternatively, evaluate whether a pure JS JSON-only constrained decoder would be small enough and reliable enough for browser usage.

Happy to hear thoughts on whether this direction makes sense before spending more time polishing it.

[nico-martin]

Little update in this:
After feedback from @xenova we decided to not bake the response_format directly into Transformers.js. The main reason is that it adds complexity to the library while only few actually use it.
Also we want to stay aligned with the Transformers python library and they focus more on providing hooks to achieve this than actually implement it into the library.

So currently I have two PRs open:

• Add Generation Extension Points for Constrained Decoding #1720: This improves the existing hooks to allow generation constraints
• Introduce @huggingface/transformers-llguidance #1721: Introduces a new package that wraps my llguidance-wasm-port to use as logits_processor/stopping_criteria in the text-generation pipe.

In the end it would be possible to use JSON and regex constraints like this:

import { pipeline } from "@huggingface/transformers";
import { LlguidanceConstraint } from "@huggingface/transformers-llguidance";

const pipe = await pipeline("text-generation", "onnx-community/gemma-4-E2B-it-ONNX", {
  device: "webgpu",
  dtype: "q4f16",
});

const { logits_processor, stopping_criteria } =
  await LlguidanceConstraint.fromResponseFormat(pipe.tokenizer, {
    type: "json_schema",
    json_schema: {
      type: "object",
      properties: {
        answer: { type: "string" },
      },
      required: ["answer"],
      additionalProperties: false,
    },
  });

const output = await pipe(
  [
    {
      role: "system",
      content: "You are a helpful assistant.",
    },
    { role: "user", content: "Answer with a short friendly greeting." },
  ],
  {
    max_new_tokens: 64,
    logits_processor,
    stopping_criteria,
  },
);

console.log("=== ANSWER ===");
console.log(JSON.parse(output[0].generated_text.pop().content));

await pipe.dispose();

This might not be the final version, but I think it's a direction we could take.

[tomayac]

Just a small naming nit: including "llguidance" in the repo name and the export name leaks a technical implementation detail that may change in the future. I'd just name it "response constraint". The API LGTM.

[nico-martin]

I see your point. However, the name was chosen deliberately. transformers-llguidance isn't per se a general JSON/regex constraint, but rather an llguidance adapter for Transformers.js. Someone could build and use a lightweight JSON constraint adapter (similar feature, different implementation). Or we could incorporate even more llguidance-based constraints into transformers-llguidance (more features, same implementation).