[Frontend] DeepSeek-V4 native OpenAI/Anthropic/Responses API + DSML tool parser - #8
[Frontend] DeepSeek-V4 native OpenAI/Anthropic/Responses API + DSML tool parser#8yhl-amd wants to merge 19 commits into
Conversation
…ool parser Serve DeepSeek-V4 (Pro/Flash) so Claude Code (Anthropic /v1/messages) and Codex CLI (OpenAI Responses /v1/responses) talk to ATOM directly, with DSML tool-call parsing shared across /v1/chat/completions, /v1/messages and /v1/responses. - tool_parser: DeepSeek-V4 DSML tool-call format (<|DSML|invoke ...>) with marker-less / self-closing / direct-JSON recovery, schema-driven type coercion and key aliases; streaming + non-streaming (alongside Qwen/GLM/MiniMax). - serving_chat: reasoning-filter + tool-call streaming for /v1/chat/completions. - serving_responses (new) + /v1/responses: OpenAI Responses translation (input/tools<->chat, SSE emitter, reasoning/function_call items). Codex compat: inject a mandatory DSML tool-format instruction and normalize shell tool name/param aliases (exec/shell_exec/... -> registered exec tool; command/script -> cmd) so Codex tool calls execute instead of erroring. - /v1/messages: pass the tool schema to the parser so tool args are typed correctly (fixes "Invalid tool parameters"); streaming UTF-8 incremental detokenization (vLLM-style sliding window) so multi-byte chars (CJK, box-drawing) aren't split into U+FFFD. - reasoning / chat_encoders: DeepSeek-V4 message encoding + reasoning tags. Builds on the abort-on-disconnect path from the previous commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes "Check Code Style with Black" CI on the DSV4 API + DSML tool parser changes (serving_responses.py, tool_parser.py, api_server.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
…mpletions streaming drains PR ROCm#1563 added the batched stream-dispatch (per-seq stream callbacks buffer their chunks into a thread-local; flush_stream_batch() drains the whole step's buffer into the per-request asyncio queues via one call_soon_threadsafe per event loop) but left the hook unwired: engine_core_mgr._flush_stream_batch_fn was only initialized to None, never resolved, and never called in the output thread. The result: streaming /v1/chat/completions and /v1/messages emitted only the initial synthetic role:assistant chunk, then hung indefinitely as the model's content chunks piled up in the thread-local buffer and never reached the per-request queue (the client timed out and the abort-on-disconnect fired). Non-streaming requests were unaffected. Fix: (a) engine_core_mgr's output thread calls self._flush_stream_batch_fn() (if set) after each step's per-seq callback dispatch; (b) the api server resolves the hook to flush_stream_batch lazily after engine init (avoids the api_server <-> engine_core_mgr import cycle). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com> (cherry picked from commit 7c2a88a07a807a7d055cef2a60351107bcd7b917)
PR ROCm#1563 only injected the mandatory DSML tool-call format instruction on the /v1/responses path (Codex), not /v1/messages (Claude Code / Anthropic). Without it, DSV4 running under Claude Code (which speaks the Anthropic /v1/messages API) does not receive the 'output ONLY a DSML tool-call block' instruction, so it emits ad-hoc/mixed tool calls instead of its native DSML format — weaker agentic behavior. Add the same inject_tool_format_instruction(openai_messages) call (gated on request.tools) that /v1/responses uses, right after anthropic_to_openai_messages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com> (cherry picked from commit 83984016fc141f14982d60cd3d0fb1637d169e4b)
…essages DSV4 (sparse indexer, index_topk=1024) cannot reliably verbatim-recall a path buried in a ~20-40k-token system prompt. Claude Code (Anthropic /v1/messages) puts the working directory in the system message but NOT in the Bash tool description, so DSV4 blind-cd's to whatever path the user named (e.g. 'cd /app/vllm-rocm && ls' on a non-existent path) instead of using the absolute path or pwd'ing first. Codex (/v1/responses) sends <cwd>...</cwd> environment_context which extract_cwd rewrites against; Claude Code has no such field (Anthropic-incompatible). Empirical A/B on the same DSV4 server (temp 0, 2 runs/condition): with the cwd in the Bash tool description DSV4 ls's the absolute path / pwd's first (0/2 blind-cd); without it DSV4 blind-cd's to the user-named path (2/2). The endpoint and the ROCm#1563 DSML-format inject are NOT the discriminating variable (held constant in the A/B); wording + cwd-presence is. Add extract_cwd_from_system (pulls 'Working directory: <path>' from the Claude Code system message) + inject_cwd_into_bash_tool (appends 'The current working directory is <path>' to the Bash tool description), and wire them on the /v1/messages path. This is a prompt-content mitigation of a model-level recall cliff, not a fix to a serving bug — the /v1/messages prompt is otherwise correctly constructed (cwd is in the system message). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com> (cherry picked from commit 6e0b326e5bbd88559f385973305fda40efb2ac63)
…onses) Three streaming generators called cleanup_streaming_request() in their finally with the default aborted=False. On client disconnect (GeneratorExit) the engine sequence was never aborted — it kept decoding to max_tokens with no consumer, wasting GPU cycles and KV-cache blocks. cleanup_streaming_request() only calls engine.core_mgr.abort_request() when aborted=True. Restore the aborted-tracking pattern already used by serving_completion.py: assume aborted=True before the stream, flip to False only after the stream reaches its normal end, and pass aborted to cleanup in the finally. Affected: stream_chat_response, stream_chat_response_fanout (serving_chat.py) and generate_responses_stream (api_server.py). Normal completion is unchanged (aborted=False -> abort is a no-op, no extra control-path broadcast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com> (cherry picked from commit a35fbf3c7f6daf6816b9e91a96a9ae688aa881f2)
Two crash sites in serving_responses.py, both reachable from model/client input
and fatal mid-stream (500 or truncated SSE):
- responses_tools_to_openai(): `tl.get("function", tl)` returns None when the
key is present with value null, so the following `fn.get("name")` raised
AttributeError. Use `tl.get("function") or tl`.
- _claude_tool_to_shell(): `int(off or 0)` / `int(lim or 200)` raised
ValueError on non-numeric offset/limit (e.g. the model emits {"offset":"top"}),
crashing the streaming handler. Wrap in try/except -> fall back to (1, 400).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: yihonglie <hyi@amd.com>
(cherry picked from commit ddc2897fcc830634824d838ca8b2464318ae220f)
extract_cwd()'s regex `<cwd>\s*([^<\s]+)\s*</cwd>` stopped at the first whitespace, truncating paths like "/home/user/My Projects/foo" to "/home/user/My" and corrupting all downstream path fix-ups. Use `[^<]+?` (keeps internal spaces/newlines, never spans a stray '<', non-greedy so it stops at the first </cwd>) and strip() the captured group. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com> (cherry picked from commit 2be8dcb46b514b70974decd2b927f6e5661c765b)
Remove extract_cwd_from_system() and inject_cwd_into_bash_tool() and their /v1/messages call site. They parsed the `Working directory: <path>` line out of Claude Code's system prompt and appended it to the Bash tool description, as a prompt-content mitigation for DSV4's recall cliff on long system prompts. Also drops the private _text_of() helper, added by the same commit and left with no callers once the two functions above are gone. Codex's <cwd>...</cwd> handling (serving_responses.extract_cwd and its _resolve_dir / _read_cmd consumers) is a separate mechanism and is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
tool_parser.py had grown to 1048 lines holding five formats, and the streaming
half was largely copy-paste: _process_/_flush_ for qwen, dsml, glm and minimax
were 57 lines each differing in exactly two — the marker lookup and which
_parse_* to call. A stale `# -- Qwen3 XML --` header left above _process_glm
recorded where the last paste happened. All five also shared one mutable
`state`/`buf` on a single dataclass, with `state`'s meaning documented for Kimi
only.
Split into a package, one module per format, named after vLLM's
tool_parsers/ layout:
tool_parser/
tool_parser.py ToolCall, ToolCallParser, BufferedMarkerParser
schema.py build_param_types / coerce_param_value /
coerce_json_or_raw
registry.py detection order + stream sniff + parse_tool_calls
stream.py ToolCallStreamParser facade
kimi_tool_parser.py
qwen3_tool_parser.py
deepseekv4_tool_parser.py
glm_tool_parser.py
minimax_tool_parser.py
Four of the five formats stream identically (buffer from a start marker, parse
the block at flush), so that strategy is written once in BufferedMarkerParser
and each format declares only START_MARKERS, HOLDBACK_CHARS and parse().
Kimi keeps its own process/flush: its token format is self-delimiting, so it
emits calls incrementally rather than buffering. GLM's and MiniMax's identical
value coercion is now one shared helper; DeepSeek-V4's is deliberately NOT
merged with it, because on a JSON-decode miss it falls back to value.strip()
where the shared one falls back to value.strip("\n").
Behavior is unchanged, including the parts that are only accidentally correct:
the detection order (documented in one place in registry.py instead of scattered
across if-branches), the streaming sniff being stricter than the non-streaming
detect, the `len(buf) > 8` heuristic, and parse_tool_calls returning unstripped
text when no format matches while every matching path strips.
Public API is unchanged — parse_tool_calls, ToolCallStreamParser and ToolCall
are re-exported from the package, so all five call sites, atomesh and the
existing tests import exactly as before. Net +59 lines: ~150 lines of duplicated
streaming logic removed, offset by per-file headers and the parser interface.
Verified: existing 13 tool-parser tests pass, plus a differential harness
comparing old vs new across 33 inputs (all five formats, malformed DSML
variants, unclosed blocks) — 226 comparisons covering parse_tool_calls and
streaming under six chunkings including one character at a time — byte-identical
after normalizing random tool-call ids.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: yihonglie <hyi@amd.com>
Remove inject_tool_format_instruction() and its _DSML_TOOL_INSTRUCTION prompt block, plus both call sites: /v1/messages (anthropic_messages) and /v1/responses (responses_endpoint). It appended a "MANDATORY tool-call format" section to the system message spelling out the DSML syntax, so a non-tuned model would emit parseable DSML instead of ad-hoc <command>/```json text. Requested removed along with the other prompt-content injections on these paths. Note this is a behavior change, not dead-code cleanup: the tool parser's DSML branch only fires on output that already looks like DSML, so models that needed the instruction to produce it may now emit tool calls the parser cannot recover. Worth re-checking DSV4-Pro tool calling on both paths. Also fixes a comment left stale by the tool_parser package split: the key-alias note referenced _parse_dsml, now DsmlParser.parse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
Keep the shared reasoning parser focused on the supported standard think tags now that MiniMax M3 compatibility is out of scope. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the shared chat-template dispatcher model-agnostic while giving DeepSeek-V4 the same synthetic tool-system message flow used by vLLM. Co-authored-by: Cursor <cursoragent@cursor.com>
Extract incremental UTF-8 detokenization and queue batching so direct and fan-out streams share one lifecycle-safe dispatch path. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the standalone Codex Responses endpoint and its tool adaptation layer so the frontend remains focused on chat and Anthropic protocols. Co-authored-by: Cursor <cursoragent@cursor.com>
… parser) into run-pr1563
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
ruff
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Do not catch blind exception: Exception
Use X | None for type annotations
Do not catch blind exception: Exception
Do not catch blind exception: Exception
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use list instead of List for type annotation
Use tuple instead of Tuple for type annotation
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
try-except-pass detected, consider logging the exception
ATOM/atom/entrypoints/openai/tool_parser/deepseekv4_tool_parser.py
Lines 157 to 158 in 3a2c0fb
Do not catch blind exception: Exception
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use tuple instead of Tuple for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use list instead of List for type annotation
Use dict instead of Dict for type annotation
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use list instead of List for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use tuple instead of Tuple for type annotation
Use tuple instead of Tuple for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use list instead of List for type annotation
Use dict instead of Dict for type annotation
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use X | None for type annotations
Use dict instead of Dict for type annotation
Use tuple instead of Tuple for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use list instead of List for type annotation
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
typing.Type is deprecated, use type instead
Use tuple instead of Tuple for type annotation
Use type instead of Type for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
typing.Dict is deprecated, use dict instead
Use X | None for type annotations
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Do not catch blind exception: Exception
Do not catch blind exception: Exception
Do not catch blind exception: Exception
Use X | None for type annotations
Use X | None for type annotations
Use X | None for type annotations
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use X | None for type annotations
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use tuple instead of Tuple for type annotation
Use tuple instead of Tuple for type annotation
- api_server.py: logger.error(..., exc_info=True) -> logger.exception(...) (G201) - streaming_dispatch.py, deepseekv4_tool_parser.py: apply Black Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
ruff
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use list instead of List for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use tuple instead of Tuple for type annotation
Use tuple instead of Tuple for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use list instead of List for type annotation
Use dict instead of Dict for type annotation
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use X | None for type annotations
Use dict instead of Dict for type annotation
Use tuple instead of Tuple for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use list instead of List for type annotation
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
typing.Type is deprecated, use type instead
Use tuple instead of Tuple for type annotation
Use type instead of Type for type annotation
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
typing.Dict is deprecated, use dict instead
Use X | None for type annotations
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Do not catch blind exception: Exception
Do not catch blind exception: Exception
Do not catch blind exception: Exception
Use X | None for type annotations
Use X | None for type annotations
Use X | None for type annotations
typing.Dict is deprecated, use dict instead
typing.List is deprecated, use list instead
typing.Tuple is deprecated, use tuple instead
Use dict instead of Dict for type annotation
Use dict instead of Dict for type annotation
Use X | None for type annotations
Use X | None for type annotations
Use tuple instead of Tuple for type annotation
Use list instead of List for type annotation
Use tuple instead of Tuple for type annotation
Use tuple instead of Tuple for type annotation
- Modernize type hints (Dict/List/Tuple/Optional -> builtins / X|None) in the new tool_parser package, chat_encoder_adapters, streaming_dispatch, and the PR-added lines of api_server/chat_encoders (UP006/UP035/UP045). - Sort api_server imports (I001). - Narrow best-effort json.loads catches to (ValueError, TypeError) (BLE001); keep one intentional catch-all with noqa (schema). - Drop redundant exception object in logger.exception (TRY401). Behavior-preserving; only files changed by this PR are touched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Modernize the module-global type hints (Dict/Optional -> dict / X|None) and mark the shared typing import noqa: UP035 (kept for the file's many pre-existing annotations). Resolves the diff-context Ruff failures. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Overview
DeepSeek-V4 native serving for the OpenAI / Anthropic / Responses APIs, plus a per-wire-format tool-parser package that parses DSML tool calls (
<｜DSML｜invoke name=...><｜DSML｜parameter ...>) into OpenAItool_calls.This mirrors ROCm/ATOM PR ROCm#1563 (
feat/dsv4-pro-openai-anthropic-responses-api), brought onto this fork so the grammar-constrained-decoding PR can stack on top of it.Key changes
atom/entrypoints/openai/tool_parser/— refactor the single-file tool parser into one module per wire format (deepseekv4_tool_parser.py,glm_,kimi_,minimax_,qwen3_), withregistry.py,schema.py,stream.py.deepseekv4_tool_parser.py— DSML → OpenAI tool_calls parsing (non-stream + streaming).atom/entrypoints/openai/protocol.py,chat_encoders.py,chat_encoder_adapters.py,streaming_dispatch.py— native OpenAI/Anthropic/Responses request/response handling.tests/entrypoints/test_chat_encoders.py,test_streaming_dispatch.py.Relationship to the grammar PR
This PR is the parsing half (DSML → tool_calls). The stacked grammar PR (
grammar-constrained-decoding) is the generation half — it constrains the model to always emit well-formed DSML. Both together fix the agentic eval (0/300 → 189/300 resolved on SWE-bench Lite).🤖 Generated with Claude Code