Skip to content

fix(prefill): keep MLX worker compute thread-affine - #165

Merged
FluffyAIcode merged 1 commit into
mainfrom
AgentMemory/mlx-prefill-worker-thread-affinity-0712
Jul 12, 2026
Merged

fix(prefill): keep MLX worker compute thread-affine#165
FluffyAIcode merged 1 commit into
mainfrom
AgentMemory/mlx-prefill-worker-thread-affinity-0712

Conversation

@FluffyAIcode

Copy link
Copy Markdown
Owner

Summary

  • construct and warm factory-backed prefill engines on the same dedicated thread that executes jobs
  • keep one MLX model per worker process and reject multi-threaded MLX job configurations
  • preserve existing injected engines for unit/service users
  • chmod generated LaunchAgent plists to 0644 before bootstrap so upgrades do not fail with launchd error 5

Root cause

The first real two-Mac request reached allens (remote_jobs +1) but MLX 0.31 rejected inference because the model/stream was created on the asyncio main thread and used by ThreadPoolExecutor: There is no Stream(gpu, 1) in current thread.

Tests

  • focused worker/orchestrator/deploy tests — 13 passed
  • platform-neutral CI selection — 816 passed, 1 skipped
  • regression proves factory construction and compute use the same non-main thread
  • Python compile, shell syntax, and diff checks pass

Live verification after merge

Redeploy allens worker, then rerun scripts/verify_remote_prefill_e2e.py; acceptance requires remote_jobs, remote_hits, and tokens_reused to increase with no fallback.

Made with Cursor

Warm the model on its dedicated job thread so MLX 0.31 streams never cross thread boundaries, and emit LaunchAgent files with permissions launchd accepts during upgrades.

Co-authored-by: Cursor <cursoragent@cursor.com>
@FluffyAIcode
FluffyAIcode merged commit 527a14f into main Jul 12, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant