From 166f012ab61e37cb59373125bc9677008ebcf627 Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Wed, 8 Jul 2026 13:39:29 +0200 Subject: [PATCH 01/12] docs: add guide for benchmarking with local tokenizer and custom JSONL dataset Adds docs/examples/custom-jsonl-dataset.md, showing how to run GuideLLM against an OpenAI-compatible endpoint using only local tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json) and a custom JSONL prompt dataset. Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 100 ++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 docs/examples/custom-jsonl-dataset.md diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md new file mode 100644 index 000000000..bb487d958 --- /dev/null +++ b/docs/examples/custom-jsonl-dataset.md @@ -0,0 +1,100 @@ +# GuideLLM Benchmark Testing with a Local Tokenizer and Custom Dataset + +Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM with a local tokenizer and a custom JSONL prompt dataset, without needing a local copy of the model itself. + +## Getting Started + +### 1. Prepare the Tokenizer Files + +The model is already deployed and running, for example on GPU, served via vLLM or OpenShift, behind an OpenAI-compatible endpoint. GuideLLM never loads or runs the model, it only talks to it over HTTP. What GuideLLM does need locally is the tokenizer, since it uses it to compute token counts for its metrics such as prompt tokens, output tokens and throughput. + +From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy only these three files: + +```bash +tokenizer.json +tokenizer_config.json +special_tokens_map.json +``` + +Place them in a local directory, e.g. `/home//mistral`. This directory will be mounted into the container and referenced as the tokenizer path. + +### 2. Prepare a Custom JSONL Dataset + +In the same directory, add your JSONL prompt file(s). Each line is a JSON object with a `prompt` field, for example: + +```json +{"prompt": "Translate the following text in full to German, French and Spanish. Provide a fluent translation of the entire text without summarizing or omitting any sentences. Respond with the translation in the same format as the original text, including line breaks and whitespace. Text: ..."} +``` + +You can split your data into multiple files by length or category if that's useful for your benchmarking scenario, for example: + +```bash +short_translation_prompts.jsonl +medium_translation_prompts.jsonl +long_translation_prompts.jsonl +``` + +After both steps, your local directory should look like this: + +```bash +ls /home//mistral +long_translation_prompts.jsonl medium_translation_prompts.jsonl short_translation_prompts.jsonl +special_tokens_map.json tokenizer_config.json tokenizer.json +``` + +### 3. Start the GuideLLM Container + +Mount the local directory into the container and drop into a shell: + +```bash +sudo docker run --rm -ti \ + -e GUIDELLM_OUTPUT_DIR="/data/mistral" \ + -e USER=guidellm \ + --volume /home//mistral:/data/mistral \ + --entrypoint bash \ + guidellm:stable +``` + +Verify the install and mounted files inside the container: + +```bash +(app-root) bash-5.2$ guidellm --version +guidellm version: 0.7.0 + +(app-root) bash-5.2$ ls /data/mistral/ +long_translation_prompts.jsonl medium_translation_prompts.jsonl short_translation_prompts.jsonl +special_tokens_map.json tokenizer_config.json tokenizer.json +``` + +______________________________________________________________________ + +## 4. Running the Benchmark + +```bash +guidellm run \ + --backend kind=openai_http,target=http:// \ + --data '{"kind":"json_file","path":"/data/mistral/long_translation_prompts.jsonl","load_kwargs":{"split":"train"}}' \ + --tokenizer '{"kind":"huggingface_auto","model":"/data/mistral"}' \ + --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \ + --profile kind=concurrent,streams=100 \ + --data-loader kind=pytorch,samples=1000 +``` + +### What Each Argument Does + +| Argument | Purpose | +|---|---| +| `--backend kind=openai_http,target=...` | Points GuideLLM at your OpenAI-compatible predictor endpoint. | +| `--data kind=json_file,path=...` | Loads prompts from the local JSONL file. | +| `--tokenizer kind=huggingface_auto,model=/data/mistral` | Loads the tokenizer from the local directory, the 3 files from step 1, instead of downloading a model. | +| `--data-column-mapper` | Maps the JSONL `prompt` field to the `text_column` GuideLLM expects internally. | +| `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | +| `--data-loader kind=pytorch,samples=1000` | Reads 1000 lines or samples from the JSONL file for the run. | + +______________________________________________________________________ + +## 5. Notes + +- The tokenizer path (`/data/mistral` in this example) only needs the 3 tokenizer files, no model weights required. +- The dataset file path and the tokenizer path can point to different directories or models if you want to mix and match, though in this example they are the same folder for convenience. +- Swap `--profile kind=concurrent,streams=N` for other GuideLLM profiles, such as `synchronous` or `throughput`, depending on the load pattern you want to test. From b065f945dc19d7edb1af015476a2b48adebe9be0 Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Thu, 9 Jul 2026 15:30:50 +0200 Subject: [PATCH 02/12] docs: fix --data-loader value to defaults To keep example so readers understand the options exists. Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index bb487d958..8387ba5f1 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -77,7 +77,7 @@ guidellm run \ --tokenizer '{"kind":"huggingface_auto","model":"/data/mistral"}' \ --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \ --profile kind=concurrent,streams=100 \ - --data-loader kind=pytorch,samples=1000 + --data-loader kind=pytorch,samples=-1 ``` ### What Each Argument Does From cd6507a58ff6ff16446a153a21c345113e708886 Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Thu, 9 Jul 2026 15:54:17 +0200 Subject: [PATCH 03/12] docs: replace obsolete env var with an example for json output file Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index 8387ba5f1..28cb9fb01 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -48,7 +48,6 @@ Mount the local directory into the container and drop into a shell: ```bash sudo docker run --rm -ti \ - -e GUIDELLM_OUTPUT_DIR="/data/mistral" \ -e USER=guidellm \ --volume /home//mistral:/data/mistral \ --entrypoint bash \ @@ -77,7 +76,8 @@ guidellm run \ --tokenizer '{"kind":"huggingface_auto","model":"/data/mistral"}' \ --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \ --profile kind=concurrent,streams=100 \ - --data-loader kind=pytorch,samples=-1 + --data-loader kind=pytorch,samples=-1 \ + --output kind=json,path=/data/mistral/benchmarks.json ``` ### What Each Argument Does @@ -90,6 +90,7 @@ guidellm run \ | `--data-column-mapper` | Maps the JSONL `prompt` field to the `text_column` GuideLLM expects internally. | | `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | | `--data-loader kind=pytorch,samples=1000` | Reads 1000 lines or samples from the JSONL file for the run. | +| `--output kind=json,path=/data/mistral/benchmarks.json` | Output file type and path. | ______________________________________________________________________ From c5b3c2caa405bcd7c86557b008d0f68ac00945ab Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Thu, 9 Jul 2026 16:02:51 +0200 Subject: [PATCH 04/12] chore: trigger pipeline Signed-off-by: Michael Stucki From e2e7c63ca3bb1c981bdcd800be68a9df400c5dba Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Thu, 9 Jul 2026 16:03:26 +0200 Subject: [PATCH 05/12] chore: trigger pipeline Signed-off-by: Michael Stucki From ed1b8ec75bae0fd368b3a142193f8a994f21f14e Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Fri, 10 Jul 2026 11:23:46 +0200 Subject: [PATCH 06/12] docs: clarify to copy necessary files if using model X Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index 28cb9fb01..17d0199fa 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -8,7 +8,8 @@ Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM wi The model is already deployed and running, for example on GPU, served via vLLM or OpenShift, behind an OpenAI-compatible endpoint. GuideLLM never loads or runs the model, it only talks to it over HTTP. What GuideLLM does need locally is the tokenizer, since it uses it to compute token counts for its metrics such as prompt tokens, output tokens and throughput. -From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy only these three files: +From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy only these three files. +This example uses a Mistral-family model; other model architectures may require additional or different files, so treat this list as a starting point rather than a universal one: ```bash tokenizer.json From ebf524bbe50f406647ccd6f66b2ce3b6b86be7ee Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Fri, 10 Jul 2026 11:31:17 +0200 Subject: [PATCH 07/12] docs: state that this example is based on local restrictions and step 3 can be ignored Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index 17d0199fa..952dd9298 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -2,6 +2,8 @@ Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM with a local tokenizer and a custom JSONL prompt dataset, without needing a local copy of the model itself. +This example runs GuideLLM from a container, reflecting our internal workflow, but that's not a requirement. A local `pip install`-based setup works the same way and skips step 3. + ## Getting Started ### 1. Prepare the Tokenizer Files From aaa9cc90c3c259621915fab75b0f03a610f2a331 Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Fri, 10 Jul 2026 11:34:29 +0200 Subject: [PATCH 08/12] docs: explicitly mention --data-loader-defaults Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index 952dd9298..977c8a333 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -90,7 +90,7 @@ guidellm run \ | `--backend kind=openai_http,target=...` | Points GuideLLM at your OpenAI-compatible predictor endpoint. | | `--data kind=json_file,path=...` | Loads prompts from the local JSONL file. | | `--tokenizer kind=huggingface_auto,model=/data/mistral` | Loads the tokenizer from the local directory, the 3 files from step 1, instead of downloading a model. | -| `--data-column-mapper` | Maps the JSONL `prompt` field to the `text_column` GuideLLM expects internally. | +| `--data-column-mapper` | Maps the JSONL `prompt` field to the `text_column` GuideLLM expects internally. Optional here since `prompt` is already a default mapping. Shown explicitly for clarity. | | `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | | `--data-loader kind=pytorch,samples=1000` | Reads 1000 lines or samples from the JSONL file for the run. | | `--output kind=json,path=/data/mistral/benchmarks.json` | Output file type and path. | From 0c26bd2d72dda1e0c87c11435339f86a1eae8c3a Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Sun, 12 Jul 2026 20:36:06 +0200 Subject: [PATCH 09/12] docs: remove all docker related parts Replaced with commands to run guidellm manually. Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 39 ++++++++------------------- 1 file changed, 11 insertions(+), 28 deletions(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index 977c8a333..697fe0fca 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -1,8 +1,7 @@ # GuideLLM Benchmark Testing with a Local Tokenizer and Custom Dataset -Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM with a local tokenizer and a custom JSONL prompt dataset, without needing a local copy of the model itself. +Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM with a local tokenizer and a custom JSONL prompt dataset, without needing a local copy of the model itself. This example runs GuideLLM via a local `pip install`. -This example runs GuideLLM from a container, reflecting our internal workflow, but that's not a requirement. A local `pip install`-based setup works the same way and skips step 3. ## Getting Started @@ -19,7 +18,7 @@ tokenizer_config.json special_tokens_map.json ``` -Place them in a local directory, e.g. `/home//mistral`. This directory will be mounted into the container and referenced as the tokenizer path. +Place them in a local directory, e.g. `/home//mistral`. ### 2. Prepare a Custom JSONL Dataset @@ -45,42 +44,26 @@ long_translation_prompts.jsonl medium_translation_prompts.jsonl short_translat special_tokens_map.json tokenizer_config.json tokenizer.json ``` -### 3. Start the GuideLLM Container - -Mount the local directory into the container and drop into a shell: - -```bash -sudo docker run --rm -ti \ - -e USER=guidellm \ - --volume /home//mistral:/data/mistral \ - --entrypoint bash \ - guidellm:stable -``` - -Verify the install and mounted files inside the container: +Verify your GuideLLM install: ```bash -(app-root) bash-5.2$ guidellm --version +guidellm --version guidellm version: 0.7.0 - -(app-root) bash-5.2$ ls /data/mistral/ -long_translation_prompts.jsonl medium_translation_prompts.jsonl short_translation_prompts.jsonl -special_tokens_map.json tokenizer_config.json tokenizer.json ``` ______________________________________________________________________ -## 4. Running the Benchmark +## 3. Running the Benchmark ```bash guidellm run \ --backend kind=openai_http,target=http:// \ - --data '{"kind":"json_file","path":"/data/mistral/long_translation_prompts.jsonl","load_kwargs":{"split":"train"}}' \ - --tokenizer '{"kind":"huggingface_auto","model":"/data/mistral"}' \ + --data '{"kind":"json_file","path":"/home//mistral/long_translation_prompts.jsonl","load_kwargs":{"split":"train"}}' \ + --tokenizer '{"kind":"huggingface_auto","model":"/home//mistral"}' \ --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \ --profile kind=concurrent,streams=100 \ --data-loader kind=pytorch,samples=-1 \ - --output kind=json,path=/data/mistral/benchmarks.json + --output kind=json,path=/home//mistral/benchmarks.json ``` ### What Each Argument Does @@ -89,16 +72,16 @@ guidellm run \ |---|---| | `--backend kind=openai_http,target=...` | Points GuideLLM at your OpenAI-compatible predictor endpoint. | | `--data kind=json_file,path=...` | Loads prompts from the local JSONL file. | -| `--tokenizer kind=huggingface_auto,model=/data/mistral` | Loads the tokenizer from the local directory, the 3 files from step 1, instead of downloading a model. | +| `--tokenizer kind=huggingface_auto,model=/home//mistral` | Loads the tokenizer from the local directory, the 3 files from step 1, instead of downloading a model. | | `--data-column-mapper` | Maps the JSONL `prompt` field to the `text_column` GuideLLM expects internally. Optional here since `prompt` is already a default mapping. Shown explicitly for clarity. | | `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | | `--data-loader kind=pytorch,samples=1000` | Reads 1000 lines or samples from the JSONL file for the run. | -| `--output kind=json,path=/data/mistral/benchmarks.json` | Output file type and path. | +| `--output kind=json,path=/home//mistral/benchmarks.json` | Output file type and path. | ______________________________________________________________________ ## 5. Notes -- The tokenizer path (`/data/mistral` in this example) only needs the 3 tokenizer files, no model weights required. +- The tokenizer path (`/home//mistral` in this example) only needs the 3 tokenizer files, no model weights required. - The dataset file path and the tokenizer path can point to different directories or models if you want to mix and match, though in this example they are the same folder for convenience. - Swap `--profile kind=concurrent,streams=N` for other GuideLLM profiles, such as `synchronous` or `throughput`, depending on the load pattern you want to test. From 160c9055c202dd088b220699bf5506c137a9c430 Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Sun, 12 Jul 2026 20:42:16 +0200 Subject: [PATCH 10/12] docs: remove defaults Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index 697fe0fca..dcdb70fb5 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -60,9 +60,7 @@ guidellm run \ --backend kind=openai_http,target=http:// \ --data '{"kind":"json_file","path":"/home//mistral/long_translation_prompts.jsonl","load_kwargs":{"split":"train"}}' \ --tokenizer '{"kind":"huggingface_auto","model":"/home//mistral"}' \ - --data-column-mapper '{"kind":"generative_column_mapper","column_mappings":{"text_column":"prompt"}}' \ --profile kind=concurrent,streams=100 \ - --data-loader kind=pytorch,samples=-1 \ --output kind=json,path=/home//mistral/benchmarks.json ``` @@ -73,9 +71,7 @@ guidellm run \ | `--backend kind=openai_http,target=...` | Points GuideLLM at your OpenAI-compatible predictor endpoint. | | `--data kind=json_file,path=...` | Loads prompts from the local JSONL file. | | `--tokenizer kind=huggingface_auto,model=/home//mistral` | Loads the tokenizer from the local directory, the 3 files from step 1, instead of downloading a model. | -| `--data-column-mapper` | Maps the JSONL `prompt` field to the `text_column` GuideLLM expects internally. Optional here since `prompt` is already a default mapping. Shown explicitly for clarity. | | `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | -| `--data-loader kind=pytorch,samples=1000` | Reads 1000 lines or samples from the JSONL file for the run. | | `--output kind=json,path=/home//mistral/benchmarks.json` | Output file type and path. | ______________________________________________________________________ From d2d57070b48c5de8de88d29e3d96b39fc77de92b Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Sun, 12 Jul 2026 20:46:12 +0200 Subject: [PATCH 11/12] docs: clarify tokenizer file list is model-specific, not universal Merge the "copy only these three files" statement with the Mistral-family caveat into one coherent paragraph, removing the contradiction between them. Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index dcdb70fb5..cfb8f2e2c 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -9,8 +9,8 @@ Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM wi The model is already deployed and running, for example on GPU, served via vLLM or OpenShift, behind an OpenAI-compatible endpoint. GuideLLM never loads or runs the model, it only talks to it over HTTP. What GuideLLM does need locally is the tokenizer, since it uses it to compute token counts for its metrics such as prompt tokens, output tokens and throughput. -From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy only these three files. -This example uses a Mistral-family model; other model architectures may require additional or different files, so treat this list as a starting point rather than a universal one: +From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy the tokenizer files required by your specific model. +This example uses a Mistral-family model, which needs only these three files. A pattern that's typical of many tokenizers, but not universal, so treat it as a starting point rather than a fixed requirement for every model: ```bash tokenizer.json From b6661cba90b4af195838df94a4aaeb3aca28da79 Mon Sep 17 00:00:00 2001 From: Michael Stucki Date: Sun, 12 Jul 2026 20:58:42 +0200 Subject: [PATCH 12/12] docs: fix markdown formatting per mdformat Signed-off-by: Michael Stucki --- docs/examples/custom-jsonl-dataset.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/examples/custom-jsonl-dataset.md b/docs/examples/custom-jsonl-dataset.md index cfb8f2e2c..9a99fff61 100644 --- a/docs/examples/custom-jsonl-dataset.md +++ b/docs/examples/custom-jsonl-dataset.md @@ -2,15 +2,13 @@ Benchmark an already-deployed OpenAI-compatible model endpoint using GuideLLM with a local tokenizer and a custom JSONL prompt dataset, without needing a local copy of the model itself. This example runs GuideLLM via a local `pip install`. - ## Getting Started ### 1. Prepare the Tokenizer Files The model is already deployed and running, for example on GPU, served via vLLM or OpenShift, behind an OpenAI-compatible endpoint. GuideLLM never loads or runs the model, it only talks to it over HTTP. What GuideLLM does need locally is the tokenizer, since it uses it to compute token counts for its metrics such as prompt tokens, output tokens and throughput. -From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy the tokenizer files required by your specific model. -This example uses a Mistral-family model, which needs only these three files. A pattern that's typical of many tokenizers, but not universal, so treat it as a starting point rather than a fixed requirement for every model: +From the model's repo, for example the HuggingFace Hub or wherever the model artifacts live, copy the tokenizer files required by your specific model. This example uses a Mistral-family model, which needs only these three files. A pattern that's typical of many tokenizers, but not universal, so treat it as a starting point rather than a fixed requirement for every model: ```bash tokenizer.json @@ -20,6 +18,8 @@ special_tokens_map.json Place them in a local directory, e.g. `/home//mistral`. +______________________________________________________________________ + ### 2. Prepare a Custom JSONL Dataset In the same directory, add your JSONL prompt file(s). Each line is a JSON object with a `prompt` field, for example: @@ -66,17 +66,17 @@ guidellm run \ ### What Each Argument Does -| Argument | Purpose | -|---|---| -| `--backend kind=openai_http,target=...` | Points GuideLLM at your OpenAI-compatible predictor endpoint. | -| `--data kind=json_file,path=...` | Loads prompts from the local JSONL file. | +| Argument | Purpose | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | +| `--backend kind=openai_http,target=...` | Points GuideLLM at your OpenAI-compatible predictor endpoint. | +| `--data kind=json_file,path=...` | Loads prompts from the local JSONL file. | | `--tokenizer kind=huggingface_auto,model=/home//mistral` | Loads the tokenizer from the local directory, the 3 files from step 1, instead of downloading a model. | -| `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | -| `--output kind=json,path=/home//mistral/benchmarks.json` | Output file type and path. | +| `--profile kind=concurrent,streams=100` | Simulates 100 concurrent "users" hitting the endpoint at once. | +| `--output kind=json,path=/home//mistral/benchmarks.json` | Output file type and path. | ______________________________________________________________________ -## 5. Notes +## 4. Notes - The tokenizer path (`/home//mistral` in this example) only needs the 3 tokenizer files, no model weights required. - The dataset file path and the tokenizer path can point to different directories or models if you want to mix and match, though in this example they are the same folder for convenience.