feat: run gateway tool loop natively on /v1/messages#131
Open
ashwing wants to merge 1 commit into
Open
Conversation
ashwing
force-pushed
the
feat/messages-gateway-tool-loop
branch
from
July 16, 2026 20:50
76d4433 to
f7a7759
Compare
ashwing
force-pushed
the
feat/messages-gateway-tool-loop
branch
6 times, most recently
from
July 17, 2026 03:18
69fffd3 to
229070d
Compare
ashwing
marked this pull request as ready for review
July 17, 2026 03:32
ashwing
requested review from
bbrowning,
franciscojavierarceo,
jiahuei,
leseb,
maralbahari,
noobHappylife,
qandrew and
tjtanaa
as code owners
July 17, 2026 03:32
ashwing
marked this pull request as draft
July 17, 2026 05:14
ashwing
force-pushed
the
feat/messages-gateway-tool-loop
branch
from
July 17, 2026 06:20
229070d to
6116f55
Compare
When an Anthropic Messages request declares a gateway-owned tool (web_search), run the server-side gateway tool loop natively against vLLM /v1/messages — execute the tool, hide it from the client, and surface only the final assistant message. Requests without a gateway-owned tool keep the transparent proxy (vllm-project#99). The loop talks Anthropic Messages end to end (no RequestPayload/ResponsePayload detour), so the client's request is forwarded to vLLM untouched and every Anthropic field is preserved. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch, name-based classification, per-call timeout, concurrent execution) via a small tool seam. - types/messages: Anthropic wire types + tool seam (tool_use<->FunctionToolCall, ToolOutput->tool_result, gateway/client classification). - executor::messages_loop: non-streaming loop (hide-the-call, feed tool_result back, round cap; client-owned tool_use returns to the client). - executor::messages_stream: streaming loop + MessagesStreamAccumulator that presents one logical message across rounds (single lifecycle, contiguous block indices, gateway tool_use suppressed, client-owned tool_use terminal) — the Anthropic-native analogue of the Responses GatewayAccumulator (vllm-project#119). - handler: routes gateway-tool requests to the loop, forwards the client's x-api-key, returns the Anthropic error envelope on failure. Tests: tool-seam + accumulator unit tests; non-streaming and streaming acceptance tests replaying live-recorded /v1/messages cassettes (single-round, parallel, sequential multi-round, no-tool-call); edge coverage for tool failure, upstream error, round cap, malformed input, and mixed client+gateway tools; handler routing tests. Validated live on G6e (Qwen3 + real You.com) via curl and a real upstream Claude Code CLI session. Implements vllm-project#115. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
ashwing
force-pushed
the
feat/messages-gateway-tool-loop
branch
from
July 17, 2026 07:36
6116f55 to
28a989c
Compare
ashwing
marked this pull request as ready for review
July 17, 2026 07:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #113; implements #115 (Stage 2).
What
Make
POST /v1/messagesrun the gateway tool loop when a request declares a gateway-owned tool (web_search): the gateway executes the tool server-side, hides it, and returns only the final assistant message. No gateway tool → unchanged transparent proxy (#99).How
The loop is native to Anthropic Messages — it talks vLLM
/v1/messagesupstream and forwards the client's request untouched, so every Anthropic field (tool_choice,stop_sequences, …) is preserved. NoRequestPayload/ResponsePayloaddetour. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch) through a small seam:types/messages/tool_seam—tool_use↔FunctionToolCall,ToolOutput→tool_result, gateway/client classification.executor::messages_loop— non-streaming loop (hide-the-call, feedtool_resultback, round cap).executor::messages_stream— streaming loop +MessagesStreamAccumulator: one logical message across rounds (single lifecycle, contiguous block indices, gatewaytool_usesuppressed). Anthropic-native analogue of the Responses accumulator ([SSE Completion - Issue 1] Unify the client-visible stream across gateway rounds (GatewayAccumulator) #119).handler— routes gateway-tool requests to the loop, forwardsx-api-key, Anthropic error envelope on failure.Scope
web_searchonly; MCP over Messages is a separate step (its wire shape isn't expressible in the Anthropic tool declaration yet). A default Claude Code session declares its own tools and proxies through — the loop engages only when a gateway-owned tool is declared.Tests
cargo fmt --check/clippy --workspace --all-targets -D warnings/cargo test --workspace— clean./v1/messagescassettes): single-round, parallel, sequential multi-round, no-tool-call, streaming.tool_result, upstream error, round cap, malformed input, mixed client+gateway tools (returns to client), proxy fallthrough.Known limitations (intentional, follow-ups)
pause_turnis surfaced, not continued — onlystop_reason: tool_usedrives another round; other stops (incl.pause_turn) return to the client. Fine as a slice; revisit if a use case needs auto-continuation.web_search-only for now.Follow-up: consolidation with the Responses loop
This adds a second gateway-tool loop alongside the existing Responses one (
run_until_gateway_tools_complete), and a second stream normalizer alongside the ResponsesGatewayAccumulator(#119/#132). That duplication is deliberate for now — the two speak different wire shapes (RequestPayload/ResponsePayloadvs raw Anthropic JSON), and per #115 the typed core shouldn't be reshaped unilaterally ahead of the Layering ADR. Tool execution is already shared (ToolRegistry::dispatchvia the tool seam); only orchestration differs.Planned as a follow-up (sequenced after the Layering ADR, coordinated with the #119/#132 work):
GatewayTurntrait + one generic round loop; Responses and Messages become impls.MAX_GATEWAY_TOOL_ROUNDS,GATEWAY_TOOL_TIMEOUT) and unify gateway-owned classification (name-based here vs structural on the Responses side) — small hygiene, can land independently.