Qwen3.6 MoE malformed XML/JSON tool calls leak as content instead of parsed tool_calls

[scouzi1966]

Summary

When serving mlx-community/Qwen3.6-35B-A3B-4bit through afm mlx with the Qwen XML/adaptive XML tool parser, some edit/run tool calls are emitted by the model inside <tool_call>...</tool_call> blocks but are not parsed into OpenAI message.tool_calls. They leak back as assistant content with tool_calls: [], so agent harnesses such as VulcanBench stop executing the intended edit/run step.

This appears to be an afm parser/template compatibility bug, not a Qwen3.6 model capability limitation:

• The same Qwen3.6 weights served through Python mlx_lm.server produced proper OpenAI tool_calls.
• Under afm, Qwen3.6 successfully emitted valid tool calls for discovery operations like list_files and read_file.
• The failures occur on larger edit/run calls where the raw output is malformed but salvageable.

Reproduction Evidence

VulcanBench comparison used the same model family and local OpenAI-compatible serving:

• afm Qwen3.6 slice: /private/tmp/VulcanBench/runs-afm-qwen36-slice
• Python MLX Qwen3.6 slice with exact same weights: /private/tmp/VulcanBench/runs-mlxpy-qwen36-slice

Observed afm Qwen3.6 failures:

• ts-querystring-bug: final edit returned raw content and no tool calls:

<tool_call>
{"name="edit_file", "arguments": {"path": "src/parse.ts", ...}}
</tool_call>

• py-topo-sort-cycle:

<tool_call>
{"function="edit_file", "path="dag/graph.py", "old_string="...", "new_string="..."}}
</tool_call>

• go-stack-pop-bug:

<tool_call>
{"function>
<name>edit_file</name>
<parameter=new_string>...</parameter>
<parameter=old_string>...</parameter>
<parameter=path>stack/pop.go</parameter>
</function>
</tool_call>

• rs-borrow-split:

<tool_call>
{"function="run_command", "arguments="cmd="find . -name '*.rs' ..."}}
</tool_call>

In each case the VulcanBench trace had tool_calls: [], so the command/edit was not executed.

Root Cause

MLXModelService.extractToolCallsFallback handles:

1. standard XML: <function=name><parameter=key>...</parameter></function>
2. XML with embedded valid JSON arguments
3. valid JSON: {"name":"func","arguments":{...}}

It did not handle the malformed Qwen3.6 hybrids above. Since the vendor parser also missed them, they remained in assistant content.

Fix Branch

Local fix branch:

fix/qwen36-malformed-toolcall-parser

The branch adds a narrow fallback parser for these Qwen3.6 malformed hybrid forms, after the existing standard XML/JSON parsers, plus regression tests for all four trace shapes.

Changed areas:

• Sources/MacLocalAPI/Models/MLXModelService.swift
• Tests/MacLocalAPITests/XMLToolCallParsingTests.swift

Verification

Fresh parser-suite verification on the fix branch:

swift test --filter XMLToolCallParsingTests

Result:

Test run with 104 tests in 1 suite passed

Follow-up

After merging the parser fix, rerun the VulcanBench Qwen3.6 slice against the rebuilt afm binary to confirm the edit/run calls are executed instead of leaked as content.

[scouzi1966]

Update after retest on branch fix/qwen36-malformed-toolcall-parser (6458cd7):

• Changed no-explicit-parser behavior so auto-detected Qwen XML uses the adaptive parser for output parsing only, but does not force the AFM XML chat-template override. This restores parity with mlx_lm.server no-parser prompting.
• Added regression coverage for parser auto-selection vs explicit template override, malformed Qwen3.6 tool-call variants, and serial generation stopping after structured tool calls.
• Verification:
  • swift test --filter XMLToolCallParsingTests: 110 tests passed.
  • swift build -c release: passed.
  • Python MLX no-parser smoke, Qwen3.6 35B, VulcanBench go-stack-pop-bug, timeout 420: functional=1.0, quality=1.0, total=0.972. Output: /private/tmp/VulcanBench/runs-mlxpy-qwen36-no-parser-timeout420-smoke/go-stack-pop-bug-d2db8150.
  • AFM MLX no-parser smoke after fix, same task/model/timeout: functional=1.0, quality=1.0, total=0.9477. Output: /private/tmp/VulcanBench/runs-afm-qwen36-no-parser-native-template-timeout420-smoke/go-stack-pop-bug-63b259f0.

Remaining finding from the four-task AFM slice: py-topo-sort-cycle produced malformed edit-file tool calls, then AFM continued a long generation after client budget expiry; subsequent tasks queued behind that stale generation and timed out. That appears to be a separate cancellation/stale-generation issue, not the no-parser template parity bug fixed here.

[scouzi1966]

Post-fix rerun (no explicit tool parser): branch fix/qwen36-malformed-toolcall-parser now includes 231c680 (Repair collapsed Qwen tool arguments). This keeps the native Qwen chat template path; AFM was started without --tool-call-parser and with --fix-tool-args --no-think.

Verification:

• swift test --filter XMLToolCallParsingTests/normalizeToolCallSplitsMalformedParameterPayload: passed.
• swift build -c release: passed (warnings only).

VulcanBench comparison using the exact same weights (mlx-community/Qwen3.6-35B-A3B-4bit):

1. Positive control, go-stack-pop-bug:

• AFM MLX: /private/tmp/VulcanBench/runs-afm-qwen36-no-parser-paramrepair-compare-isolated/go-stack-pop-bug-c000d586; functional=1.0, quality=1.0, total=0.9459. Trace shows clean tool calls including edit_file(new_string, old_string, path) and run_tests().
• Python MLX: /private/tmp/VulcanBench/runs-mlxpy-qwen36-paramrepair-compare-isolated/go-stack-pop-bug-a666ab8a; functional=1.0, quality=1.0, total=0.9585.

2. Reproducer, py-topo-sort-cycle:

• AFM MLX: /private/tmp/VulcanBench/runs-afm-qwen36-no-parser-paramrepair-isolated/py-topo-sort-cycle-f19429e0; functional=0.0, total=0.1426, budget_exceeded=true. The recorded tool calls before timeout were valid (list_files, read_file); no malformed parameter payload reached the harness. It hit VulcanBench's 300s task budget during step-5 LLM response.
• Python MLX: /private/tmp/VulcanBench/runs-mlxpy-qwen36-paramrepair-compare-isolated/py-topo-sort-cycle-14272571; functional=0.0, total=0.1419. It read the files, then returned explanatory text with no tool call, producing no diff. Python MLX server also logged a parser warning for a truncated/missing function.

Current classification:

• The original AFM bug around native-template handling and collapsed single-parameter tool arguments is fixed by 6458cd7 + 231c680 and covered by tests.
• py-topo-sort-cycle still fails for both backends with the same weights, but not for the same reason: AFM times out waiting for a later response, while Python MLX returns no edit. This points to model/backend behavior for that task rather than the repaired malformed-argument parser path.
• There is still a separate AFM runtime issue worth tracking: non-streaming MLX generation does not appear to cancel promptly when VulcanBench abandons a timed-out request; earlier multi-task slices were contaminated by one stale long generation blocking subsequent tasks.

[scouzi1966]

Follow-up fix pushed on branch fix/qwen36-malformed-toolcall-parser at af75e41.

What changed:

• AFM MLX now matches Python mlx_lm.server when neither request nor CLI sets max_tokens: default is 512 instead of 4096.
• Native Qwen tool prompts no longer use swift-jinja's compact/sorted tojson for tool specs. When no explicit --tool-call-parser is used, AFM patches the native template to consume a Python/HF-style tool JSON string with the same spacing shape as mlx_lm.server.
• The internal __python_json__ field is only added on that patched native-template path, so it should not leak into ordinary/unpatched templates or explicit parser templates.

Verification:

• swift test --filter IssueRegressionTests --filter XMLToolCallParsingTests: 119 tests passed.
• swift build -c release: passed.

VulcanBench rerun with exact same weights: /Volumes/Crucial4TB/models/vesta-test-cache/mlx-community/Qwen3.6-35B-A3B-4bit, no explicit AFM tool parser, no Apple on-device model.

AFM fixed release:

• py-topo-sort-cycle: /private/tmp/VulcanBench/runs-afm-qwen36-default512-templatepatch-compare/py-topo-sort-cycle-60dcfe2e => functional=0.0, total=0.1341. No timeout. First request prompt tokens now 1028 and max_tokens=512.
• go-stack-pop-bug: /private/tmp/VulcanBench/runs-afm-qwen36-default512-templatepatch-compare/go-stack-pop-bug-c4b3f0ad => functional=1.0, quality=1.0, total=0.9572.

Python mlx_lm.server baseline with same weights and --chat-template-args '{"enable_thinking":false}':

• py-topo-sort-cycle: /private/tmp/VulcanBench/runs-mlxpy-qwen36-nothink-512-rerun-after-afmfix/py-topo-sort-cycle-826fb46b => functional=0.0, total=0.142. Python log also shows first request prompt processing 1028/1028 and warns: failed to parse tool call; tool text likely truncated mid-generation.
• go-stack-pop-bug: /private/tmp/VulcanBench/runs-mlxpy-qwen36-nothink-512-rerun-after-afmfix/go-stack-pop-bug-e41e9c2c => functional=1.0, quality=1.0, total=0.9544.

Current conclusion: the AFM-specific issues found here were real and are fixed by this branch: default-token mismatch, native prompt spacing mismatch, and earlier malformed/collapsed parameter repair. The remaining py-topo-sort-cycle failure is not currently isolated to AFM; fixed AFM and Python MLX now match the key prompt/default behavior and both fail because Qwen emits/truncates a long edit tool call at the 512-token default instead of producing an executable parsed tool call.

[scouzi1966]

Additional follow-up pushed on fix/qwen36-malformed-toolcall-parser: 337e3a1 (Cancel abandoned non-streaming MLX generations).

While trying to expand from the 2-task comparison to the 12-task VulcanBench compare subset, AFM exposed a separate runtime issue: after a long/budget-exceeded non-streaming run, the next request could be accepted by the HTTP server but sit behind the previous decode holding the serial model lock. The streaming path already checked Task.isCancelled inside generation; the non-streaming path did not.

Patch:

• Non-streaming MLX generation now calls Task.checkCancellation() before entering the generation loop.
• It also checks Task.isCancelled inside the loop and throws CancellationError, invalidating cache through the existing generation-error path and releasing the model lock.

Verification:

• swift test --filter IssueRegressionTests --filter XMLToolCallParsingTests: 119 tests passed.
• swift build -c release: passed.

Partial broad-suite evidence:

• Pre-patch 12-task compare loop reached py-retry-refactor, which budget-exceeded after 300s: /private/tmp/VulcanBench/runs-afm-qwen36-compare-suite-af75e41/py-retry-refactor-b6db5cb4, functional=0.0, total=0.1124, budget_exceeded=true. The following py-jsonpointer request was accepted but did not begin prefill while the abandoned decode/shutdown case was still active.
• Post-patch rerun reached py-topo-sort-cycle: /private/tmp/VulcanBench/runs-afm-qwen36-compare-suite-cancel-af75e41/py-topo-sort-cycle-680db05d, functional=0.0, total=0.1494, budget_exceeded=true. The server no longer required force-kill on shutdown, but runtime throughput degraded badly in that session, so I stopped the 12-task run rather than treating it as valid parity evidence.

Current status: prompt/default/tool-argument parity fixes are pushed and still verified. The remaining broad-suite parity claim is not proven yet; the expanded run found a cancellation/stale-lock AFM bug, now fixed, plus a separate degraded-throughput runtime condition that needs a clean rerun before claiming full VulcanBench parity.

[scouzi1966]

Retest update after attempting to replace the plain MLXLMCommon.generate(...) loop with explicit generateTask(...) handling, still without passing --tool-call-parser.

What was verified:

• swift build -c release: passed.
• swift test --filter IssueRegressionTests --filter XMLToolCallParsingTests: passed, 119 tests.

AFM VulcanBench transition rerun against Qwen3.6 35B MoE:

• Command shape: AFM mlx server on 127.0.0.1:9999, --no-streaming --fix-tool-args --no-think, no explicit tool parser.
• Output dir: /private/tmp/VulcanBench/runs-afm-qwen36-transition-gentask.
• py-retry-refactor-e41f6269: functional=0.0, total=0.1238, budget_exceeded=true, duration_s=300.065, total_tokens=11734.
• Immediately following py-jsonpointer-10d612e9: functional=0.0, total=0.1642, budget_exceeded=true, duration_s=300.065, total_tokens=0.
• py-jsonpointer trace shows it timed out on its first llm_request without receiving any model tokens. This means the first attempted generateTask/await cleanup was not sufficient; the prior request can still monopolize the model lock after client timeout.

Additional attempted mitigation:

• I tried a bounded cancellation grace path (AFM_GENERATION_CANCEL_GRACE_SECONDS, default 2s) so abandoned generation does not wait forever inside await generationTask.value.
• Build and the same 119 focused tests passed.
• A fresh VulcanBench rerun in /private/tmp/VulcanBench/runs-afm-qwen36-transition-bounded-cancel became unusably degraded on the first request (1094 prompt tokens took 48.192s, generation 81 tokens took 75.45s; about 22.7 prompt tok/s and 1.1 gen tok/s versus normal ~800/~90). I stopped that run; it produced only /private/tmp/VulcanBench/runs-afm-qwen36-transition-bounded-cancel/py-retry-refactor-089834fe/trace.jsonl, no summary. Stopping the server exited with code 139.

Conclusion: prompt/tool parity fixes remain good, but the cancellation/stale-generation bug is not fixed yet. The remaining AFM-specific issue is lower-level MLX generation cancellation/shutdown: cancellation is only observed between iterator.next() calls, and the task can be stuck inside an uninterruptible MLX/GPU step. Python MLX with the same weights budget-exceeds this task too, but does not leave the next request at zero tokens in the same way.

[houleixx]

Raw tool-call inspection would be useful here.

ccglass can show the exact response body from an OpenAI-compatible/local model provider, including content, tool_calls, tool results, and streamed chunks.

Project: https://github.com/jianshuo/ccglass

For this bug, it could help confirm whether the malformed <tool_call>...</tool_call> appears directly in the model response, or whether it is introduced/
lost during parsing into OpenAI-style tool_calls.