Add vLLM offline backend with micro-batching support - #736

Open

maryamtahhan wants to merge 29 commits into

vllm-project:mainfrom

maryamtahhan:feat/vllm-offline-batching-backend

Commits on Aug 7, 2026

Add vLLM Offline Backend for batch processing

maryamtahhan
and
claude
committed
fcdf58a
Fix tokenizer access path for vLLM V1 LLM engine

maryamtahhan
and
claude
committed
1f48b85
Fix formatting, metrics, and mdformat issues

maryamtahhan
and
claude
committed
99235e3
Defer vLLM engine creation to first request and reset CPU affinity

maryamtahhan
and
claude
committed
f8c0b79
Wire vLLM RequestStateStats metrics and fix batch lock contention

maryamtahhan
and
claude
committed
547510a
Address review: move CPU affinity to common module, use _shutting_down, call shutdown

maryamtahhan
and
claude
committed
d486f15
Fix concurrency hazards and clean up batch request dataclass

maryamtahhan
and
claude
committed
ddbda0c
Add unit tests for vLLM offline backend

maryamtahhan
and
claude
committed
73fb806
Fix shutdown race, add batch_timeout, and finish test coverage

maryamtahhan
and
claude
committed
45a6eef
Replace guide example with throughput profile and remove getattr from metrics

maryamtahhan
and
claude
committed
ff01dc2
Reset asyncio locks in process_startup to fix forked worker hang

maryamtahhan
and
claude
committed
cb946cb
Preload engine in forked workers to fix 0-completion benchmark

maryamtahhan
and
claude
committed
31080b3
Polish engine lifecycle docs and strengthen lock-reset test

maryamtahhan
and
claude
committed
251a06e
Address PR review comments for vllm_offline backend

maryamtahhan
and
claude
committed
238843c
Fix partial-metrics crash in _wire_vllm_metrics and address review nits

maryamtahhan
and
claude
committed
f079b07
Detect offline worker processes via parent_process() for engine preload

maryamtahhan
and
cursoragent
committed
52218a9
Fix benchmark progress panel stacking during vLLM offline worker init.

maryamtahhan
and
claude
committed
067dbc0
Suppress vLLM worker startup output and fix offline batch waiter hang.

maryamtahhan
and
claude
committed
9a2b7b8
Address low-severity gaps: fd leak, markers, and missing tests.

maryamtahhan
and
claude
committed
26b5dff
Fix offline batch waiter hangs and spawn worker pickling.

maryamtahhan
and
cursoragent
committed
38ce340
Fix quality check failures: mypy union-attr, ruff lint, and ruff format

maryamtahhan
and
claude
committed
f7eab23
Fix remaining ruff lint failures blocking quality CI.

maryamtahhan
and
cursoragent
committed
34e27e5
Remove Windows guard from suppress_worker_stdio.

maryamtahhan
and
claude
committed
c80c827
Simplify prepare_vllm_benchmark_logging to env vars only.

maryamtahhan
and
claude
committed
a9172f6
Fall back to plain prompt when model has no chat template.

maryamtahhan
and
claude
committed
3b0cb8e
Harden prepare_vllm_benchmark_logging with handler lowering and defensive vLLM reconfigure.

maryamtahhan
and
claude
committed
8930494
Fix ruff lint errors in prepare_vllm_benchmark_logging hardening.

maryamtahhan
and
claude
committed
1b463e1
Lower vllm.* child logger levels during benchmark logging setup.

maryamtahhan
and
cursoragent
committed
a6330c8
Address PR #736 review nits for VLLMOfflineBackend.

maryamtahhan
and
claude
committed
74f9e1e

Navigation Menu

Uh oh!

Add vLLM offline backend with micro-batching support - #736

Add vLLM offline backend with micro-batching support#736
maryamtahhan wants to merge 29 commits into
vllm-project:mainfrom
maryamtahhan:feat/vllm-offline-batching-backend

Commits on Aug 7, 2026

Add vLLM Offline Backend for batch processing

Fix tokenizer access path for vLLM V1 LLM engine

Fix formatting, metrics, and mdformat issues

Defer vLLM engine creation to first request and reset CPU affinity

Wire vLLM RequestStateStats metrics and fix batch lock contention

Address review: move CPU affinity to common module, use _shutting_down, call shutdown

Fix concurrency hazards and clean up batch request dataclass

Add unit tests for vLLM offline backend

Fix shutdown race, add batch_timeout, and finish test coverage

Replace guide example with throughput profile and remove getattr from metrics

Reset asyncio locks in process_startup to fix forked worker hang

Preload engine in forked workers to fix 0-completion benchmark

Polish engine lifecycle docs and strengthen lock-reset test

Address PR review comments for vllm_offline backend

Fix partial-metrics crash in _wire_vllm_metrics and address review nits

Detect offline worker processes via parent_process() for engine preload

Fix benchmark progress panel stacking during vLLM offline worker init.

Suppress vLLM worker startup output and fix offline batch waiter hang.

Address low-severity gaps: fd leak, markers, and missing tests.

Fix offline batch waiter hangs and spawn worker pickling.

Fix quality check failures: mypy union-attr, ruff lint, and ruff format

Fix remaining ruff lint failures blocking quality CI.

Remove Windows guard from suppress_worker_stdio.

Simplify prepare_vllm_benchmark_logging to env vars only.

Fall back to plain prompt when model has no chat template.

Harden prepare_vllm_benchmark_logging with handler lowering and defensive vLLM reconfigure.

Fix ruff lint errors in prepare_vllm_benchmark_logging hardening.

Lower vllm.* child logger levels during benchmark logging setup.

Address PR #736 review nits for VLLMOfflineBackend.