[KV Offloading] Add basic offloading metrics by Srinivasoo7 · Pull Request #1 · Srinivasoo7/vllm

Srinivasoo7 · 2026-06-17T19:33:24Z

Summary

add vllm:kv_offload_lookup_delay_seconds
add vllm:kv_offload_allocation_failure
add vllm:kv_offload_cpu_allocation_size
add focused scheduler and CPU manager coverage for the new metrics

github-actions · 2026-06-17T19:33:46Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

Signed-off-by: mayuyuace <qiming1.zhang@intel.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…m-project#46450) Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com> Co-authored-by: Or Ozeri <oro@il.ibm.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…-project#47125) Signed-off-by: Bugen Zhao <i@bugenzhao.com>

…ons (vllm-project#45960) Signed-off-by: Seiji Eicher <seiji@anyscale.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…up-size/TP mismatch (vllm-project#46230) (vllm-project#46236) Signed-off-by: Arsalan Shakil <shakil.arsalan@yahoo.com>

…tion sampling (vllm-project#46781) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>

…set (vllm-project#47007) Signed-off-by: jperezde <jperezde@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…tion (vllm-project#45918) Signed-off-by: tc-mb <tianchi_cai@icloud.com>

…Us)` (vllm-project#47157) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

…iers (vllm-project#46301) Signed-off-by: Igor Margulis <igor.margulis@intel.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mgoin <mgoin64@gmail.com>

Signed-off-by: yewentao256 <zhyanwentao@126.com>

…m-project#47050) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: ziminghuang <ziminghuang@inferact.ai>

…t#47140) Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…prevent corrupted output (vllm-project#43637) Signed-off-by: fangyuchu <fangyuchu@qq.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…-project#47065) Signed-off-by: charlifu <charlifu@amd.com>

…lm-project#47105) Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…in non-streaming tool parsing (vllm-project#46875) Signed-off-by: Ben Browning <bbrownin@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Signed-off-by: Anshika Ojha <anshikao@nvidia.com> Signed-off-by: Rishi Puri <riship@nvidia.com> Signed-off-by: Rishi Puri <puririshi98@berkeley.edu> Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com> Co-authored-by: Anshika Ojha <215760622+ojhaanshika@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>

… in macos + cpu (vllm-project#47457) Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>

vllm-project#46168) Signed-off-by: lcheng <lcheng321@gatech.edu> Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai>

…ant (vllm-project#43994) Signed-off-by: Sean Chen <seachen@redhat.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…ability_threshold semantics (vllm-project#45965) Signed-off-by: Nathaniel McVicar <namcvica@microsoft.com> Signed-off-by: Nathaniel McVicar <Nathaniel.McVicar@microsoft.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

…43328) Signed-off-by: Andrii Skliar <askliar@nvidia.com> Co-authored-by: Andrii Skliar <askliar@nvidia.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Co-authored-by: yy-fighting <23518844576@qq.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

…-project#41652) Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: mgoin <mgoin64@gmail.com>

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: sherryC41 <sherry.c.c41@gmail.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>

…P8 KV (vllm-project#47276) Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com> Co-authored-by: nnyrhila <niko.nyrhila@amd.com>

Signed-off-by: Andreas Karatzas <akaratza@amd.com>

…tion + Mistral/Granite parsers) (vllm-project#47550) Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>

… Chat/Completions APIs (vllm-project#46768) Signed-off-by: Nicholas Edelman <nedelman@nvidia.com> Signed-off-by: Simon Mo <simon@inferact.ai> Co-authored-by: Simon Mo <simon@inferact.ai> Co-authored-by: GPT-5.5 <noreply@cursor.com>

Signed-off-by: yisheng <yi.sheng@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

Signed-off-by: reidliu41 <reid201711@gmail.com>

Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com>

…e groups (vllm-project#45159) Signed-off-by: jialoop-git <joane8913456@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Nicolas Hug <contact@nicolas-hug.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…ct#47581) Signed-off-by: reidliu41 <reid201711@gmail.com>

…ect#47787) Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>

vllm-project#45672) Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>

…project#47148) Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

…project#47685) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

…m-project#45352) Signed-off-by: HumphreySun98 <humphreysun98@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Srinivasoo7 force-pushed the offloading-labeled-metrics branch 2 times, most recently from a03fac4 to dac20e2 Compare June 21, 2026 16:16

Srinivasoo7 force-pushed the offloading-basic-metrics branch 4 times, most recently from 755bbcf to 4ea575d Compare June 28, 2026 16:40

Srinivasoo7 force-pushed the offloading-basic-metrics branch from 4ea575d to 781ebe8 Compare June 29, 2026 15:12

mayuyuace and others added 22 commits June 30, 2026 21:34

[XPU] C++ implementation for get_memory_info (vllm-project#47134)

91055ef

Signed-off-by: mayuyuace <qiming1.zhang@intel.com>

[CI/Build] Bump PyNvVideoCodec version (vllm-project#47139)

ab80b3d

Forward fix nightly errors from vllm-project#44589 (vllm-project#47151)

62c7d80

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[KV Offload] Pass ScheduleEndContext to on_schedule_end hook (vll…

0fc2512

…m-project#46450) Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com> Co-authored-by: Or Ozeri <oro@il.ibm.com>

Remove more unnecessary load_weights methods (vllm-project#47058)

1ab9522

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[Rust Frontend] Simplify unit tests with shared TestTokenizer (vllm…

7a327f0

…-project#47125) Signed-off-by: Bugen Zhao <i@bugenzhao.com>

[Bugfix] Seed RayExecutorV2 TCPStore port by DP rank to avoid collisi…

ded6676

…ons (vllm-project#45960) Signed-off-by: Seiji Eicher <seiji@anyscale.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

[Bugfix][Quant] Raise actionable error instead of bare assert for gro…

00ebf19

…up-size/TP mismatch (vllm-project#46230) (vllm-project#46236) Signed-off-by: Arsalan Shakil <shakil.arsalan@yahoo.com>

[Model Runner V2][Spec Decode] Implement block verification for rejec…

db808b3

…tion sampling (vllm-project#46781) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>

fix(security): bound tokenizer work when explicit truncation_side is …

c231d1f

…set (vllm-project#47007) Signed-off-by: jperezde <jperezde@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

[Bugfix] MiniCPM-V 4.6: fix grid rows/cols swap in placeholder genera…

7cf7cbc

…tion (vllm-project#45918) Signed-off-by: tc-mb <tianchi_cai@icloud.com>

[CI][Bugfix] Fix `Hybrid SSM NixlConnector PD prefix cache test (2 GP…

dc148dc

…Us)` (vllm-project#47157) Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>

[Refactor] Remove dead minimax allreduce rms kernel (vllm-project#46842)

9e84ec8

Signed-off-by: yewentao256 <zhyanwentao@126.com>

[BugFix] Gate MRV2 mixed sparse-MLA warmup on max_num_seqs > 1 (vll…

fcaa84e

…m-project#47050) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: ziminghuang <ziminghuang@inferact.ai>

[Platform] Replace torch.cuda.Event with torch.Event (vllm-projec…

e840f0d

…t#47140) Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[ROCm][CI] Move PyTorch Compilation Unit Tests to MI300(gfx942) (vllm…

f41e8dd

…-project#47065) Signed-off-by: charlifu <charlifu@amd.com>

[XPU] Support ZE_AFFINITY_MASK passthrough in xpu_disagg_acc_test (vl…

7a341fa

…lm-project#47105) Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>

[CI] Move distributed small LM eval to B200 (vllm-project#47048)

27d5f78

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

[Parser][Bugfix] Ensure tool call or other special tokens don't leak …

25671cb

…in non-streaming tool parsing (vllm-project#46875) Signed-off-by: Ben Browning <bbrownin@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

WindChimeRan and others added 30 commits July 6, 2026 08:59

[macOS][CPU][Installation] Fix the broken installation of vllm 0.24.0…

3ee9eea

… in macos + cpu (vllm-project#47457) Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>

[Kernel][Helion][1/N] Add Helion kernel for silu_and_mul_per_block_qu…

9fde043

…ant (vllm-project#43994) Signed-off-by: Sean Chen <seachen@redhat.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Merge branch 'main' into offloading-basic-metrics

b560f8f

Enable B12x backend for non-gated MoEs (like Nemotron) (vllm-project#…

b1384f5

…43328) Signed-off-by: Andrii Skliar <askliar@nvidia.com> Co-authored-by: Andrii Skliar <askliar@nvidia.com>

[CI] Fix some errors on main (vllm-project#47726)

ae098ab

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

[Bugfix]Fix DeepSeek-V4 fp8_ds_mla KV cache reshape (vllm-project#47716)

04adc88

Co-authored-by: yy-fighting <23518844576@qq.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

[Quantization] add humming moe backend to all dense/moe oracles (vllm…

d891b9b

…-project#41652) Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: mgoin <mgoin64@gmail.com>

[Bugfix] Fix dp mtp hang (vllm-project#40589)

567a784

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: sherryC41 <sherry.c.c41@gmail.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>

[Bugfix][ROCm] Fix memory access fault in AITER MLA backend for DPA+F…

482e552

…P8 KV (vllm-project#47276) Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com> Co-authored-by: nnyrhila <niko.nyrhila@amd.com>

[ROCm][CI] Adding Rust parity (vllm-project#47478)

8484ca5

Signed-off-by: Andreas Karatzas <akaratza@amd.com>

[ROCm][CI][Bugfix] Fix flaky parallel tool-call streaming (test asser…

5769a73

…tion + Mistral/Granite parsers) (vllm-project#47550) Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com>

Merge branch 'main' into offloading-basic-metrics

aeac42d

[XPU] Fix PP accuracy on XPU device (vllm-project#47253)

69f3150

Signed-off-by: yisheng <yi.sheng@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>

Merge branch 'main' into offloading-basic-metrics

99014a0

[Bugfix] [Quantization] Fix loading for CT DSV2 (vllm-project#47780)

445321f

[Rust Frontend] Add DeepSeek V3.2 roundtrip fixture (vllm-project#47619)

a46c932

Signed-off-by: reidliu41 <reid201711@gmail.com>

feat(cpu): add CPU support for Mamba ShortConv (vllm-project#35059)

9dd2465

Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com>

Merge branch 'main' into offloading-basic-metrics

1e4b720

fix(distributed): propagate distributed_timeout_seconds to NCCL devic…

a4f019f

…e groups (vllm-project#45159) Signed-off-by: jialoop-git <joane8913456@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add TorchCodec as a video decoding backend (vllm-project#46609)

700e882

Signed-off-by: Nicolas Hug <contact@nicolas-hug.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Rust Frontend] Avoid extra copies for multimodal tensors (vllm-proje…

39a1d32

…ct#47581) Signed-off-by: reidliu41 <reid201711@gmail.com>

[Rust Frontend] Stamp arrival_time at the frontend entry (vllm-proj…

34e6dfc

…ect#47787) Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>

[Perf] Bound DiffusionGemma sampler transient via request-tiled logits (

c64c356

vllm-project#45672) Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>

[UX] Add model_class_overrides for development and debugging (vllm-…

32ab064

…project#47148) Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

[ROCm] Align mixed encoder-decoder KV cache views in V2 runner (vllm-…

2f71b2b

…project#47685) Signed-off-by: Andreas Karatzas <akaratza@amd.com>

[Bugfix] Forward callable hf_overrides to the draft model config (vll…

cbe9c40

…m-project#45352) Signed-off-by: HumphreySun98 <humphreysun98@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

Merge branch 'main' into offloading-basic-metrics

9d4f8ce

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KV Offloading] Add basic offloading metrics#1

[KV Offloading] Add basic offloading metrics#1
Srinivasoo7 wants to merge 882 commits into
offloading-labeled-metricsfrom
offloading-basic-metrics

Srinivasoo7 commented Jun 17, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

Srinivasoo7 commented Jun 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Srinivasoo7 commented Jun 17, 2026 •

edited by github-actions Bot

Loading