Skip to content

feat: run gateway tool loop natively on /v1/messages#131

Open
ashwing wants to merge 1 commit into
vllm-project:mainfrom
ashwing:feat/messages-gateway-tool-loop
Open

feat: run gateway tool loop natively on /v1/messages#131
ashwing wants to merge 1 commit into
vllm-project:mainfrom
ashwing:feat/messages-gateway-tool-loop

Conversation

@ashwing

@ashwing ashwing commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Part of #113; implements #115 (Stage 2).

What

Make POST /v1/messages run the gateway tool loop when a request declares a gateway-owned tool (web_search): the gateway executes the tool server-side, hides it, and returns only the final assistant message. No gateway tool → unchanged transparent proxy (#99).

How

The loop is native to Anthropic Messages — it talks vLLM /v1/messages upstream and forwards the client's request untouched, so every Anthropic field (tool_choice, stop_sequences, …) is preserved. No RequestPayload/ResponsePayload detour. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch) through a small seam:

  • types/messages/tool_seamtool_useFunctionToolCall, ToolOutputtool_result, gateway/client classification.
  • executor::messages_loop — non-streaming loop (hide-the-call, feed tool_result back, round cap).
  • executor::messages_stream — streaming loop + MessagesStreamAccumulator: one logical message across rounds (single lifecycle, contiguous block indices, gateway tool_use suppressed). Anthropic-native analogue of the Responses accumulator ([SSE Completion - Issue 1] Unify the client-visible stream across gateway rounds (GatewayAccumulator) #119).
  • handler — routes gateway-tool requests to the loop, forwards x-api-key, Anthropic error envelope on failure.

Scope

web_search only; MCP over Messages is a separate step (its wire shape isn't expressible in the Anthropic tool declaration yet). A default Claude Code session declares its own tools and proxies through — the loop engages only when a gateway-owned tool is declared.

Tests

  • cargo fmt --check / clippy --workspace --all-targets -D warnings / cargo test --workspace — clean.
  • Unit: tool seam + streaming accumulator.
  • Acceptance (replay live-recorded /v1/messages cassettes): single-round, parallel, sequential multi-round, no-tool-call, streaming.
  • Edges: tool failure → error tool_result, upstream error, round cap, malformed input, mixed client+gateway tools (returns to client), proxy fallthrough.
  • Live G6e (Qwen3 + real You.com): curl scenarios + a real upstream Claude Code CLI session — tool executed server-side and hidden, single streaming lifecycle, no raw-Responses leakage.

Known limitations (intentional, follow-ups)

  • pause_turn is surfaced, not continued — only stop_reason: tool_use drives another round; other stops (incl. pause_turn) return to the client. Fine as a slice; revisit if a use case needs auto-continuation.
  • MCP over Messages is out of scope — the Anthropic tool declaration doesn't express MCP's server config/dynamic names yet; gateway classification is web_search-only for now.

Follow-up: consolidation with the Responses loop

This adds a second gateway-tool loop alongside the existing Responses one (run_until_gateway_tools_complete), and a second stream normalizer alongside the Responses GatewayAccumulator (#119/#132). That duplication is deliberate for now — the two speak different wire shapes (RequestPayload/ResponsePayload vs raw Anthropic JSON), and per #115 the typed core shouldn't be reshaped unilaterally ahead of the Layering ADR. Tool execution is already shared (ToolRegistry::dispatch via the tool seam); only orchestration differs.

Planned as a follow-up (sequenced after the Layering ADR, coordinated with the #119/#132 work):

  • Extract a GatewayTurn trait + one generic round loop; Responses and Messages become impls.
  • A shared stream-normalizer contract for the two accumulators (single lifecycle, contiguous indices, hide-the-call).
  • Hoist the shared limits (MAX_GATEWAY_TOOL_ROUNDS, GATEWAY_TOOL_TIMEOUT) and unify gateway-owned classification (name-based here vs structural on the Responses side) — small hygiene, can land independently.

@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch from 76d4433 to f7a7759 Compare July 16, 2026 20:50
@ashwing ashwing changed the title feat: run gateway tool loop on /v1/messages (non-streaming) feat: run gateway tool loop on /v1/messages Jul 16, 2026
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch 6 times, most recently from 69fffd3 to 229070d Compare July 17, 2026 03:18
@ashwing
ashwing marked this pull request as ready for review July 17, 2026 03:32
@ashwing
ashwing marked this pull request as draft July 17, 2026 05:14
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch from 229070d to 6116f55 Compare July 17, 2026 06:20
@ashwing ashwing changed the title feat: run gateway tool loop on /v1/messages feat: run gateway tool loop natively on /v1/messages Jul 17, 2026
When an Anthropic Messages request declares a gateway-owned tool (web_search),
run the server-side gateway tool loop natively against vLLM /v1/messages —
execute the tool, hide it from the client, and surface only the final
assistant message. Requests without a gateway-owned tool keep the transparent
proxy (vllm-project#99).

The loop talks Anthropic Messages end to end (no RequestPayload/ResponsePayload
detour), so the client's request is forwarded to vLLM untouched and every
Anthropic field is preserved. It reuses only the protocol-neutral tool layer
(ToolRegistry::dispatch, name-based classification, per-call timeout,
concurrent execution) via a small tool seam.

- types/messages: Anthropic wire types + tool seam (tool_use<->FunctionToolCall,
  ToolOutput->tool_result, gateway/client classification).
- executor::messages_loop: non-streaming loop (hide-the-call, feed tool_result
  back, round cap; client-owned tool_use returns to the client).
- executor::messages_stream: streaming loop + MessagesStreamAccumulator that
  presents one logical message across rounds (single lifecycle, contiguous
  block indices, gateway tool_use suppressed, client-owned tool_use terminal) —
  the Anthropic-native analogue of the Responses GatewayAccumulator (vllm-project#119).
- handler: routes gateway-tool requests to the loop, forwards the client's
  x-api-key, returns the Anthropic error envelope on failure.

Tests: tool-seam + accumulator unit tests; non-streaming and streaming
acceptance tests replaying live-recorded /v1/messages cassettes (single-round,
parallel, sequential multi-round, no-tool-call); edge coverage for tool
failure, upstream error, round cap, malformed input, and mixed client+gateway
tools; handler routing tests. Validated live on G6e (Qwen3 + real You.com) via
curl and a real upstream Claude Code CLI session.

Implements vllm-project#115.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
@ashwing
ashwing force-pushed the feat/messages-gateway-tool-loop branch from 6116f55 to 28a989c Compare July 17, 2026 07:36
@ashwing
ashwing marked this pull request as ready for review July 17, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant