Add example: benchmarking with local tokenizer and custom JSONL dataset by miglis · Pull Request #915 · vllm-project/guidellm

miglis · 2026-07-08T11:42:40Z

Summary

benchmarking with local tokenizer and custom JSONL dataset

Adds docs/examples/custom-jsonl-dataset.md, showing how to run GuideLLM against an OpenAI-compatible endpoint using only local tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json) and a custom JSONL prompt dataset.

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes code generated or substantially modified by an AI agent
Includes tests generated or substantially modified by an AI agent

git log

commit 166f012
Author: Michael Stucki michael.stucki99@gmail.com
Date: Wed Jul 8 13:39:29 2026 +0200

docs: add guide for benchmarking with local tokenizer and custom JSONL dataset

Adds docs/examples/custom-jsonl-dataset.md, showing how to run
GuideLLM against an OpenAI-compatible endpoint using only local tokenizer
files (tokenizer.json, tokenizer_config.json, special_tokens_map.json)
and a custom JSONL prompt dataset.

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit b065f94
Author: Michael Stucki michael.stucki99@gmail.com
Date: Thu Jul 9 15:30:50 2026 +0200

docs: fix --data-loader value to defaults

To keep example so readers understand the options exists.

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit cd6507a
Author: Michael Stucki michael.stucki99@gmail.com
Date: Thu Jul 9 15:54:17 2026 +0200

docs: replace obsolete env var with an example for json output file

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit c5b3c2c
Author: Michael Stucki michael.stucki99@gmail.com
Date: Thu Jul 9 16:02:51 2026 +0200

chore: trigger pipeline

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit e2e7c63
Author: Michael Stucki michael.stucki99@gmail.com
Date: Thu Jul 9 16:03:26 2026 +0200

chore: trigger pipeline

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit ed1b8ec
Author: Michael Stucki michael.stucki99@gmail.com
Date: Fri Jul 10 11:23:46 2026 +0200

docs: clarify to copy necessary files if using model X

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit ebf524b
Author: Michael Stucki michael.stucki99@gmail.com
Date: Fri Jul 10 11:31:17 2026 +0200

docs: state that this example is based on local restrictions and step 3 can be ignored

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit aaa9cc9
Author: Michael Stucki michael.stucki99@gmail.com
Date: Fri Jul 10 11:34:29 2026 +0200

docs: explicitly mention --data-loader-defaults

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit 0c26bd2
Author: Michael Stucki michael.stucki99@gmail.com
Date: Sun Jul 12 20:36:06 2026 +0200

docs: remove all docker related parts

Replaced with commands to run guidellm manually.

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit 160c905
Author: Michael Stucki michael.stucki99@gmail.com
Date: Sun Jul 12 20:42:16 2026 +0200

docs: remove defaults

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit d2d5707
Author: Michael Stucki michael.stucki99@gmail.com
Date: Sun Jul 12 20:46:12 2026 +0200

docs: clarify tokenizer file list is model-specific, not universal

Merge the "copy only these three files" statement with the Mistral-family
caveat into one coherent paragraph, removing the contradiction between them.

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

commit b6661cb
Author: Michael Stucki michael.stucki99@gmail.com
Date: Sun Jul 12 20:58:42 2026 +0200

docs: fix markdown formatting per mdformat

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

Signed-off-by: Michael Stucki michael.stucki99@gmail.com

mergify · 2026-07-08T11:43:14Z

Hi @miglis, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this.

dbutenhof

There's some overlap with the practice_on_vllm_simulator.md example -- for example it talks about downloading a tokenizer to run locally, although it doesn't specify the particular files.

Generally, more concrete examples are good. My comments are generally in areas where you're either too abstract or unnecessarily over-specified (like using unnecessary options to specify defaults or values inconsistent with your example)...

mergify · 2026-07-09T14:03:35Z

Hi @miglis, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this.

…L dataset Adds docs/examples/custom-jsonl-dataset.md, showing how to run GuideLLM against an OpenAI-compatible endpoint using only local tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json) and a custom JSONL prompt dataset. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

To keep example so readers understand the options exists. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

… 3 can be ignored Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

dbutenhof

This is improving -- sorry, but still a few more problems.

dbutenhof · 2026-07-10T15:54:29Z

By the way, you need to fix your file formatting:

run markdown formatter...................................................Failed
- hook id: mdformat
- files were modified by this hook

So, when you're making further changes, git add the updated file to your local git index, then

$ pip install pre-commit
$ pre-commit install
$ pre-commit run

It will give you a message like the CI's message above ... only in this case, the modified files are in your local branch. Add the changes to your git index, then you can re-run pre-commit run to verify that there are no remaining problems before committing the changes.

Replaced with commands to run guidellm manually. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

Merge the "copy only these three files" statement with the Mistral-family caveat into one coherent paragraph, removing the contradiction between them. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

dbutenhof

One somewhat awkward sentence could be clarified.

And while I'm basically happy about removing the extra steps of the containerized deployment and the unnecessary default options, I also feel just a little bad that I pushed you away from what you were trying to do. (Just a little: but... "sorry".)

SkiHatDuckie

I've got no complaints.

dbutenhof · 2026-07-15T17:58:26Z

@Mergifyio queue

mergify · 2026-07-15T17:58:30Z

dbutenhof · 2026-07-15T18:39:26Z

@Mergifyio queue

mergify · 2026-07-15T18:39:29Z

queue

☑️ Command `queue` ignored because it is already running from a previous command.

The substance of Sam's comment has been met.

dbutenhof · 2026-07-15T18:42:04Z

@Mergifyio refresh

mergify · 2026-07-15T18:42:10Z

refresh

✅ Pull request refreshed

miglis force-pushed the docs/jsonl-example branch from 7de165a to ef4aa02 Compare July 8, 2026 11:44

dbutenhof reviewed Jul 8, 2026

View reviewed changes

Comment thread docs/examples/custom-jsonl-dataset.md

Comment thread docs/examples/custom-jsonl-dataset.md Outdated

Comment thread docs/examples/custom-jsonl-dataset.md Outdated

Comment thread docs/examples/custom-jsonl-dataset.md

sjmonson previously requested changes Jul 8, 2026

View reviewed changes

Comment thread docs/examples/custom-jsonl-dataset.md Outdated

miglis force-pushed the docs/jsonl-example branch from 5366e5f to a693064 Compare July 9, 2026 13:55

miglis added 5 commits July 9, 2026 16:03

docs: fix --data-loader value to defaults

b065f94

To keep example so readers understand the options exists. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

docs: replace obsolete env var with an example for json output file

cd6507a

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

chore: trigger pipeline

c5b3c2c

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

chore: trigger pipeline

e2e7c63

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

miglis force-pushed the docs/jsonl-example branch from 1f841f9 to e2e7c63 Compare July 9, 2026 14:04

miglis requested a review from sjmonson July 9, 2026 14:05

dbutenhof requested a review from jaredoconnell July 9, 2026 17:16

miglis added 3 commits July 10, 2026 11:23

docs: clarify to copy necessary files if using model X

ed1b8ec

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

docs: state that this example is based on local restrictions and step…

ebf524b

… 3 can be ignored Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

docs: explicitly mention --data-loader-defaults

aaa9cc9

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

dbutenhof requested changes Jul 10, 2026

View reviewed changes

Comment thread docs/examples/custom-jsonl-dataset.md Outdated

Comment thread docs/examples/custom-jsonl-dataset.md Outdated

Comment thread docs/examples/custom-jsonl-dataset.md Outdated

miglis and others added 5 commits July 12, 2026 20:13

Merge branch 'vllm-project:main' into docs/jsonl-example

4ce166f

docs: remove all docker related parts

0c26bd2

Replaced with commands to run guidellm manually. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

docs: remove defaults

160c905

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

docs: clarify tokenizer file list is model-specific, not universal

d2d5707

Merge the "copy only these three files" statement with the Mistral-family caveat into one coherent paragraph, removing the contradiction between them. Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

docs: fix markdown formatting per mdformat

b6661cb

Signed-off-by: Michael Stucki <michael.stucki99@gmail.com>

miglis requested a review from dbutenhof July 12, 2026 19:01

dbutenhof approved these changes Jul 13, 2026

View reviewed changes

Comment thread docs/examples/custom-jsonl-dataset.md

Merge branch 'vllm-project:main' into docs/jsonl-example

1992107

dbutenhof requested a review from SkiHatDuckie July 15, 2026 15:05

SkiHatDuckie approved these changes Jul 15, 2026

View reviewed changes

mergify Bot added the queued label Jul 15, 2026

mergify Bot added a commit that referenced this pull request Jul 15, 2026

Merge of #915

6be74c3

mergify Bot mentioned this pull request Jul 15, 2026

merge queue: checking main (3e3f9df) and #915 together #932

Closed

35 tasks

mergify Bot merged commit 6c63a06 into vllm-project:main Jul 15, 2026
12 checks passed

mergify Bot removed the queued label Jul 15, 2026

Uh oh!

Conversation

miglis commented Jul 8, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Use of AI

git log

Uh oh!

mergify Bot commented Jul 8, 2026

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented Jul 9, 2026

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbutenhof commented Jul 10, 2026

Uh oh!

dbutenhof left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SkiHatDuckie left a comment

Choose a reason for hiding this comment

Uh oh!

dbutenhof commented Jul 15, 2026

Uh oh!

mergify Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

dbutenhof commented Jul 15, 2026

Uh oh!

mergify Bot commented Jul 15, 2026

☑️ Command queue ignored because it is already running from a previous command.

Uh oh!

dbutenhof commented Jul 15, 2026

Uh oh!

mergify Bot commented Jul 15, 2026

✅ Pull request refreshed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

miglis commented Jul 8, 2026 •

edited by github-actions Bot

Loading

mergify Bot commented Jul 15, 2026 •

edited

Loading

☑️ Command `queue` ignored because it is already running from a previous command.