Skip to content

feat(distributed): orchestrate same-model prefill workers and KV memory tier - #160

Merged
cursor[bot] merged 11 commits into
mainfrom
AgentMemory/prefill-worker-orchestration-2815
Jul 12, 2026
Merged

feat(distributed): orchestrate same-model prefill workers and KV memory tier#160
cursor[bot] merged 11 commits into
mainfrom
AgentMemory/prefill-worker-orchestration-2815

Conversation

@FluffyAIcode

@FluffyAIcode FluffyAIcode commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Architecture

Implements ADR 0017: one primary Mac owns RuntimeService/session/decode; peer Macs either load the exact same model and execute prefill-only jobs (PREFILL_COMPUTE) or devote RAM to immutable KV snapshots (PREFILL_CACHE). Remote state is imported once before decode; no remote RPC occurs in the autoregressive loop. DFlash/f_theta/fused speculative decode are legacy research, not product completeness criteria.

Implementation

  • PrefillWorkerService: idempotent bounded jobs, status/cancel, deadlines, cancellation race guards, TTL/history cap and co-located cache leases.
  • MLXPrefillComputeEngine, worker node and headless launchd installer.
  • Gossip cards advertise exact model/tenant/sink/window/geometry compatibility, queue tokens/load/TPS/free RAM.
  • Primary cost model selects cache import, remote compute or local prefill; rendezvous placement bounds replication.
  • Mandatory local fallback for every lookup/job/lease/fetch/checksum/decompress/import failure.
  • Snapshot token-boundary/hash/retained-token/continuation-logit validation; bounded chunk assembly and decompression.
  • zlib framing, SHA-256, fleet PSK request HMAC, replay window and tenant-HMAC prefix hashes.
  • Worker telemetry in network API; operator runbook and ADR 0017.

Runner/CI hardening discovered during validation

  • Integration now uses direct path triggers instead of the non-recursive GITHUB_TOKEN label race.
  • Fixed listener recovery detection, stale locks and official-service/direct fallback.
  • Added a headless system LaunchDaemon watchdog + runner healthcheck.
  • Bootstraps git-lfs before checkout (reused worktree retained an LFS post-checkout hook).
  • Resolves the real .venv-mac interpreter and stable verifier path.
  • Separates Transformers-5 MLX gate from a PEP-668-safe legacy Qwen venv.

Validation — all green

  • ✅ Real Mac MLX continuation-equivalence: 1 passed (local prefill vs snapshot import preserves exact logits/argmax/retained tokens/one decode step).
  • ✅ Full Mac M4 integration: pass, 2m13s.
  • ✅ Linux unit + 100% coverage: pass, 2m14s.
  • ✅ TypeScript SDK, proto lint/format/stub drift, package import and Docker: pass.
  • ✅ Distributed modules locally: 167 passed, 1 MLX-only skipped, 100% line coverage including prefill_cache_runtime.py.
  • ✅ Authenticated loopback orchestration: gossip worker -> remote prefill -> cache lease -> chunked fetch -> validated primary import.
Open in Web Open in Cursor 

cursoragent and others added 3 commits July 11, 2026 15:09
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
cursoragent and others added 8 commits July 11, 2026 15:11
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…espace

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
@cursor
cursor Bot merged commit 6026d2e into main Jul 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants