Explore converting existing llama.cpp GBNF 'structured output' config to use the SOTA llguidance format (which now also powers OpenAI API JSON Schema, custom tools/etc)

[0xdevalias]

Interesting anecdote, llama.cpp has support for the rust-based version of guidance built in now, and is also what is used by the OpenAI API for their JSON Schemas now too:

• https://github.com/guidance-ai/llguidance

  • Low-level Guidance (llguidance)

  • 2025-05-20 LLGuidance shipped in OpenAI for JSON Schema

  • 2025-02-01 integration merged into llama.cpp (b4613)

    • support for llguidance grammars ggml-org/llama.cpp#10224
    • https://github.com/ggml-org/llama.cpp/blob/master/docs/llguidance.md

      • LLGuidance supports JSON Schemas and arbitrary context-free grammars (CFGs) written in a variant of Lark syntax. It is very fast and has excellent JSON Schema coverage but requires the Rust compiler, which complicates the llama.cpp build process.

      • There are no new command-line arguments or modifications to common_params. When enabled, grammars starting with %llguidance are passed to LLGuidance instead of the current llama.cpp grammars. Additionally, JSON Schema requests (e.g., using the -j argument in llama-cli) are also passed to LLGuidance.

      • For your existing GBNF grammars, you can use gbnf_to_lark.py script to convert them to LLGuidance Lark-like format.

        • https://github.com/guidance-ai/llguidance/blob/main/docs/syntax.md

          • LLGuidance supports a variant of syntax used by Python Lark parsing toolkit. We also provide a gbnf_to_lark.py script to convert from GBNF format used in llama.cpp. These makes it easier to get started with a new grammar, and provide a familiar syntax, however neither is a drop-in replacement for Lark or GBNF.

Originally posted by @0xdevalias in #6 (comment)

See also:

• humanify/src/plugins/local-llm-rename/gbnf.ts

  Lines 1 to 73 in 7beba2d

  	export class Gbnf {
  	rule: string;
  	genStart: number;
  	genEnd?: number;
  	constructor(rule: string, genStart: number, genEnd?: number) {
  	this.rule = rule;
  	this.genStart = genStart;
  	this.genEnd = genEnd;
  	}
  	toString() {
  	return this.rule;
  	}
  	parseResult(result: string) {
  	return result.slice(this.genStart, this.genEnd);
  	}
  	}
  	export function gbnf(
  	strings: TemplateStringsArray,
  	...values: (string | RegExp)[]
  	) {
  	const numRegexes = values.filter((value) => value instanceof RegExp).length;
  	if (numRegexes > 1) {
  	throw new Error("Only one variable per rule is supported");
  	}
  	let rule = "root ::=";
  	for (let i = 0; i < strings.length; i++) {
  	rule += ` "${strings[i].replaceAll('"', '\\"')}"`;
  	const value = values[i];
  	if (value instanceof RegExp) {
  	rule += ` ` + value.source;
  	} else if (typeof value == "string") {
  	rule += ` "${value.replaceAll('"', '\\"')}"`;
  	} else {
  	// Undefined
  	}
  	}
  	if (numRegexes === 0) {
  	return new Gbnf(rule, 0, undefined);
  	}
  	let startVar = 0;
  	let endVar = 0;
  	let isPastRegex = false;
  	for (let i = 0; i < strings.length; i++) {
  	if (isPastRegex) {
  	endVar -= strings[i].length;
  	} else {
  	startVar += strings[i].length;
  	}
  	const value = values[i];
  	if (value instanceof RegExp) {
  	isPastRegex = true;
  	} else if (typeof value == "string") {
  	if (isPastRegex) {
  	endVar -= value.length;
  	} else {
  	startVar += value.length;
  	}
  	} else {
  	// Undefined
  	}
  	}
  	return new Gbnf(rule, startVar, endVar);
  	}

• humanify/src/plugins/openai/openai-rename.ts

  Lines 62 to 79 in 7beba2d

  	response_format: {
  	type: "json_schema",
  	json_schema: {
  	strict: true,
  	name: "rename",
  	schema: {
  	type: "object",
  	properties: {
  	newName: {
  	type: "string",
  	description: `The new name for the variable/function called \`${name}\``
  	}
  	},
  	required: ["newName"],
  	additionalProperties: false
  	}
  	}
  	}

• https://platform.openai.com/docs/guides/function-calling#custom-tools
  • https://platform.openai.com/docs/guides/function-calling#context-free-grammars

    • Context-free grammars

      A context-free grammar (CFG) is a set of rules that define how to produce valid text in a given format. For custom tools, you can provide a CFG that will constrain the model's text input for a custom tool.

      You can provide a custom CFG using the grammar parameter when configuring a custom tool. Currently, we support two CFG syntaxes when defining grammars: lark and regex.

    • Grammars are specified using a variation of Lark. Model sampling is constrained using LLGuidance.

    • We recommend using the Lark IDE to experiment with custom grammars.

• https://guidance-ai.github.io/llguidance/llg-go-brrr

  • LLGuidance: Making Structured Outputs Go Brrr

• https://github.com/guidance-ai/guidance-ts

[jehna]

Nice. Seems that llama.cpp needs an extra build time flag for this, need to investigate if this is on by default:

llama.cpp - available via -DLLAMA_LLGUIDANCE=ON

[jehna]

And AFAIK this is only used internally by OpenAI's JSON Schema output, no way to use it via API directly