PR-B3 (ADR 0008 Phase B): Generate server-streaming RPC + greedy session-aware decoding

[FluffyAIcode]

✅ Mac M4 §9 integration test report — landed (commit 16196be)

First Phase-B PR with mandatory ADR 0008 §9 Mac M4 report; carve-out
NOT applicable. All 5 artifacts pass.

Artifact	Result
pr-b3-mac-generator-tests-1780323650.json	31 / 31 passed, 100.00 % on inference_engine/session/generator.py (63 / 63 lines)
pr-b3-mac-grpc-tests-1780323650.json	39 / 39 passed (PR-B1 + B2 + B3 cumulative), 100.00 % on inference_engine/server/grpc_app.py (88 / 88 lines)
pr-b3-mac-grpc-runtime-smoke-1780323650.json	10 / 10 (PR-B1 contract regression: AppendTokens + Generate stay UNIMPLEMENTED with bare Servicer)
pr-b3-mac-grpc-appender-smoke-1780323650.json	10 / 10 (PR-B2 contract regression: AppendTokens reachable, Generate still UNIMPLEMENTED)
pr-b3-mac-grpc-generator-smoke-1780323650.json	10 / 10 (PR-B3 new)
Host	macOS-26.5-arm64 / Python 3.13.12 / grpcio 1.81.0

Generator smoke per-step (PR-B3 wire behavior on Apple Silicon)

#	Step	Expected	Observed	Time
1	CreateSession	ok	ok	0.98 ms
2	AppendTokens (cold prefill)	ok	ok	0.48 ms
3	Generate (max_tokens=3, no EOS)	ok	ok	0.63 ms
4	Generate (EOS triggers, max_tokens=10)	ok	ok	1.01 ms
5	Generate (no AppendTokens prior)	INVALID_ARGUMENT	INVALID_ARGUMENT	0.57 ms
6	Generate (max_tokens=0)	INVALID_ARGUMENT	INVALID_ARGUMENT	0.33 ms
7	Generate (temperature=0.7, non-greedy rejected)	INVALID_ARGUMENT	INVALID_ARGUMENT	0.31 ms
8	Generate (unknown session)	NOT_FOUND	NOT_FOUND	0.34 ms
9	Generate (truncated state → truncated frame)	ok	ok	1.50 ms
10	Generate (no coordinator wired)	UNIMPLEMENTED	UNIMPLEMENTED	0.72 ms

10 / 10 step times in 0.31 – 1.50 ms, similar order to Linux (0.27 – 1.55 ms) — confirms gRPC streaming behavior is platform-equivalent on Apple Silicon.

Design choice: greedy first, speculative later

PR-B3 ships greedy decoding only. Speculative-decoding integration
(DLM proposer + AR verifier rejection sampling — Kakeya's
distinguishing feature) is reserved for a later PR. Two reasons:

1. Wire contract is algorithm-agnostic — GenerateResponse oneof on
   token_id / done / truncated works for any decoding algorithm.
   Switching to speculative is a coordinator-internal refactor, not a
   protocol break.
2. Scope discipline — PR-B3 is already the first PR triggering §9;
   adding speculative would couple two large changes. Greedy is the
   minimum surface that satisfies §9; speculative integration can land
   independently with its own integration test pass.

temperature / top_p / top_k MUST be in their greedy no-op
defaults. Setting any non-default raises ValueError → INVALID_ARGUMENT
per §2.10 "no graceful degradation" — the runtime refuses to silently
downgrade.

Three deliverables

1. inference_engine/session/generator.py (new, ~240 lines)

Symbol	Role
TokenEvent / HistoryTruncatedEvent / DoneEvent	Frozen dataclass payloads
STOP_REASON_*	String constants matching runtime_pb2.GenerateDone.StopReason
GenerationCoordinator(store, verifier)	Sync iterator yielding GenerateEvent per step

Per-step contract:

next_token = int(torch.argmax(verifier.next_token_logits).item())
block_logits = verifier.forward_block([next_token])
verifier.commit_or_truncate(forwarded=1, accepted=1)
verifier.next_token_logits = block_logits[-1].clone()
session.cached_token_sequence = list(verifier.cached_token_sequence)
store.append_tokens(session_id, [next_token])     # extends history + INV-1
store.record_position_advance(session_id, verifier.next_global_position)  # INV-2
yield TokenEvent(token_id=next_token)
if next_token in eos_set: yield DoneEvent(STOP_REASON_EOS, ...); return

2. RuntimeServiceServicer.Generate (new)

Path	gRPC status
Servicer constructed without generation_coordinator	UNIMPLEMENTED
SessionNotFoundError	NOT_FOUND
ValueError	INVALID_ARGUMENT
InvariantViolation	FAILED_PRECONDITION
context.cancelled() mid-stream	Final GenerateDone(STOP_REASON_CANCELLED), then return
Success	Stream of token_id (+ optional initial truncated), terminated by done

HistoryTruncated frame: emitted at most once per call, before any
token_id, when the session is in sink+window-truncated state at
Generate's start. Matches runtime.proto contract exactly.

3. INV-3 byte-exact unit tests via FakeVerifier

TestDeterminism.test_two_runs_with_same_history_produce_same_tokens
drives two parallel sessions with identical history through Generate
and asserts byte-identical token streams under greedy decoding.

The 31-test generator suite covers every code path on generator.py
(63 / 63 lines) including:

• 9 validation paths (max_tokens, sampling params, no-AppendTokens)
• 2 invariant paths (INV-1 / INV-2 propagation through Generate)
• 3 EOS paths
• 3 HistoryTruncated paths
• 1 byte-exact determinism path

Files

Status	Path
add	inference_engine/session/generator.py (~240 lines)
add	tests/inference_engine/session/test_generator.py (31 tests)
add	scripts/smoke_grpc_generator.py (10-scenario smoke)
add	scripts/review_pr_b3_on_mac.sh (5-artifact reviewer)
add	results/platform-tests/pr-b3-mac-*-1780323650.* (Mac M4 evidence: 5 JSON + 2 junit.xml + 2 coverage.xml)
mod	inference_engine/server/grpc_app.py (+119: Generate RPC + factory keyword + cancellation)
mod	inference_engine/session/__init__.py (re-exports)
mod	tests/inference_engine/server/test_grpc_app.py (+11 tests including direct cancellation invocation)

Linux verification

PYTHONPATH=. pytest <Linux CI gate set> --cov-fail-under=100
629 passed (was 587 + 31 generator + 11 grpc Generate)
TOTAL  1521 stmts  100.00 % coverage

Reviewer checklist

• Mac M4 evidence on branch: 5 JSON artifacts pushed (commit 16196be), all expected counts match.
• Greedy contract enforced: temperature/top_p/top_k non-default → INVALID_ARGUMENT. Verified Mac M4 step Engine E2: OpenAI-compatible HTTP API with SSE streaming #7.
• Cancellation tested: direct Servicer invocation with FakeContext confirms STOP_REASON_CANCELLED Done frame.
• HistoryTruncated emitted exactly once per call, before any token_id. Verified Mac M4 step Engine E4: continuous batching scheduler (admission control + fair queuing) #9.
• INV-3 byte-exact under greedy: TestDeterminism passes.
• No new MLX runtime paths beyond per-token forward_block + commit_or_truncate (these are NOT new — they were exercised by AppendTokens in PR-B2; Generate uses them in a per-token loop).
• PooledVerifier / inference_engine.memory/ untouched.

Next PR

PR-B4: ship sdks/python/ — kakeya.Client, kakeya.Session,
100% unit-test coverage against an in-process gRPC server. First
external-facing API surface; Linux-only path.