ConcurrentInferenceServer

A multithreaded C++20 inference-serving layer over llama.cpp: a bounded, thread-safe request queue with backpressure, a worker pool of independent decoding contexts, live token streaming, and per-request inference instrumentation (TTFT, inter-token latency, throughput).

What this is / isn't

• Compute kernel: llama.cpp / ggml, unmodified, linked as a subproject.
• This project: the concurrent serving layer around it — request scheduling, a bounded queue with an explicit backpressure policy, a worker pool where each thread owns its own llama_context, deterministic cooperative shutdown, and production-style inference metrics.

The point is the systems engineering, not the model. No tensor math is reimplemented.

Why this exists

llama.cpp already ships a production server (llama-server) that is OpenAI-compatible, supports parallel decoding, and is maintained by a large community. Nobody should run this instead of llama-server for real work — theirs is battle-tested and full-featured, and this one deliberately isn't.

This is a from-scratch implementation built to understand and demonstrate the serving layer itself: request scheduling, backpressure, worker/context lifecycle, concurrency scaling, and (on the roadmap) in-flight batching. The value is in being able to build and reason about these internals — the same ones that sit inside any production inference server — not in replacing a tool that already does the job well. Where measurements or design choices here line up with how llama-server behaves, that's the point: the goal is to understand why a real server is built the way it is.

Benchmarks

Model: Qwen2.5-1.5B-Instruct (1.54 B params), three quant levels from the same repo. Hardware: Intel i7-9700F (8 cores / 8 threads, AVX2), 64 GB RAM, NVIDIA RTX 2060 SUPER (8 GB), Windows 10 / MSVC. Workload: 16 requests × 128 tokens each, greedy sampling. On CPU the total compute-thread budget is held constant at 8 (workers × threads = 8) so the rows are apples-to-apples.

CPU (AVX2)

Quant	Model RAM	Workers×Threads	Aggregate tok/s	TTFT p50	TTFT p95	ITL mean
Q4_K_M	0.91 GB (935 MiB)	1×8	21.2	150.8 ms	220.9 ms	46.2 ms/tok
		2×4	25.0	273.1 ms	375.4 ms	78.5 ms/tok
		4×2	26.8	504.2 ms	567.5 ms	144.2 ms/tok
		8×1	26.4	1096.1 ms	1109.4 ms	269.3 ms/tok
Q8_0	1.53 GB (1565 MiB)	1×8	15.2	168.5 ms	254.8 ms	65.0 ms/tok
		2×4	16.6	306.4 ms	389.0 ms	119.2 ms/tok
		4×2	17.3	593.8 ms	786.2 ms	222.5 ms/tok
		8×1	17.4	1301.4 ms	1404.7 ms	439.8 ms/tok
F16	2.88 GB (2945 MiB)	1×8	8.5	223.5 ms	343.3 ms	116.9 ms/tok
		2×4	8.8	428.4 ms	619.8 ms	226.6 ms/tok
		4×2	9.4	565.0 ms	932.9 ms	424.4 ms/tok
		8×1	9.3	697.3 ms	1030.3 ms	842.5 ms/tok

(Model RAM is the reported weight buffer; add ~0.4 GB per worker for KV cache + compute buffers at --n-ctx 4096. Bold = best aggregate throughput per quant.)

GPU (CUDA 13, RTX 2060 SUPER 8 GB, --gpu-layers 99, single device)

Quant	Workers	Aggregate tok/s	TTFT p50	TTFT p95	ITL mean
Q4_K_M	1	96.9	19.6 ms	44.0 ms	10.2 ms/tok
Q4_K_M	2	131.0	32.6 ms	109.8 ms	15.0 ms/tok
Q4_K_M	4	133.4	51.8 ms	185.0 ms	29.4 ms/tok

(All layers offloaded to the RTX 2060 SUPER. The box's second GPU — a Pascal Quadro P620 — is excluded via CUDA_VISIBLE_DEVICES=0, since the CUDA build targets sm_75 only. Q8_0/F16 GPU rows are omitted: at 1.5B every quant fits in 8 GB VRAM and decode is compute-bound, so the concurrency story is identical.)

Interpretation (CPU). Quantization dominates throughput: Q4_K_M sustains ~2.8× the tokens/sec of F16 (26.8 vs 9.4 peak) at under a third of the memory (0.91 vs 2.88 GB), with Q8_0 sitting in between (~17 tok/s, 1.53 GB). CPU decode is memory-bandwidth-bound — every token streams the entire weight set, so fewer bytes per weight translates almost linearly into more tokens/sec, making Q4_K_M the clear default for CPU serving. Worker scaling is deliberately modest here: with the thread budget pinned at 8, adding workers only overlaps each other's memory stalls, so aggregate throughput creeps up ~25% from 1→4 workers (21→27 tok/s on Q4_K_M) and then flattens/dips at 8 — the workers are dividing the same cores and the same memory bus, so there is no linear speedup to be had. What does change sharply is per-request latency: TTFT p50 climbs from ~150 ms to ~1100 ms and ITL from 46 to 269 ms/token as workers go 1→8, because each request now runs on fewer threads and contends for the bus. The lesson: on a fixed CPU budget, concurrency buys throughput stability and better core utilization, not scaling, and it trades latency to get it.

Interpretation (GPU). Offloading to the 2060 SUPER lifts single-worker throughput from ~21 tok/s (CPU) to ~97 tok/s and drops ITL from 46 to 10 ms/token — a 3–4× win, since the GPU's memory bandwidth dwarfs the CPU's. The concurrency behavior is the real lesson: 1→2 workers adds ~35% (96.9→131.0 tok/s) because a single 1.5B decode stream doesn't saturate the device and two streams overlap to fill it — but 2→4 workers is essentially flat (131→133 tok/s) while latency keeps climbing (ITL 10→15→29 ms/token, TTFT p95 44→185 ms). That plateau is the point: every worker context shares the same GPU, so once the device is the bottleneck, adding parallel contexts buys no throughput and only piles on latency and VRAM. Scaling past it requires decoding multiple sequences in a single GPU step — continuous / in-flight batching, the next roadmap item — not N independent contexts contending for one device. (This contrasts with the CPU, where each worker can be handed its own physical cores.)

Build (Windows / MSVC)

Requires CMake 3.21+, a C++20 MSVC toolchain, and Git. The first configure clones and builds the pinned llama.cpp tag (a few minutes); the llama/ggml (and, for the CUDA build, ggml-cuda) DLLs are copied next to the executable automatically.

There are two build variants. They live in separate build directories (build/ and build-cuda/) so they can coexist — build either or both.

CPU-only (default, portable)

Works on any machine — no GPU or CUDA Toolkit required. Uses the ggml CPU backend (AVX2 on this i7-9700F).

cmake -B build
cmake --build build --config Release --parallel

Linux/macOS build identically. This is the recommended default.

CUDA (optional, GPU-accelerated)

Requires the NVIDIA CUDA Toolkit (nvcc) and a driver new enough for that toolkit's runtime (CUDA 13.x needs an r580+ driver). -DGGML_CUDA=ON switches ggml's compute backend from CPU to CUDA.

cmake -B build-cuda -DGGML_CUDA=ON
cmake --build build-cuda --config Release --parallel

Run with --gpu-layers 99 to offload all layers to the GPU. On this box the 8 GB RTX 2060 SUPER comfortably fits 1–3B models at any quant (Q4/Q8/F16).

• Target architecture. Configure with -DCMAKE_CUDA_ARCHITECTURES=75 for the Turing (sm_75) 2060 SUPER. The default (native) auto-detects all installed GPUs, which fails here because CUDA 13 dropped Pascal (the box's Quadro P620 is sm_61).
• Multi-GPU boxes. The build targets one architecture; select the matching device at runtime with CUDA_VISIBLE_DEVICES=0 (the 2060 SUPER) so llama.cpp doesn't try to also use the P620.

Linux builds identically. macOS uses Metal (on by default) instead of CUDA.

Get a model

Download any GGUF model into models/ (git-ignored). A small (1–3B) instruct model in a few quant levels is ideal for benchmarking:

models/
  model-Q4_K_M.gguf
  model-Q8_0.gguf
  model-F16.gguf

Run

Single request, streamed live (Milestone 1):

.\build\Release\inference_server.exe --model models\model-Q4_K_M.gguf `
  --prompt "Explain what a mutex is." --n-predict 128

Concurrent load, aggregate stats (Milestone 2):

.\build\Release\inference_server.exe --model models\model-Q4_K_M.gguf `
  --workers 4 --requests 16 --n-predict 128

Run --help for all options.

Benchmarking

Sweep concurrency and quantization to fill the table above:

foreach ($w in 1,2,4,8) {
  .\build\Release\inference_server.exe --model models\model-Q4_K_M.gguf `
    --workers $w --requests ($w * 4) --n-predict 128
}

Repeat per quant level. Aggregate throughput is total generated tokens over wall-clock time; TTFT/ITL are reported per request and summarized as p50/p95.

Design notes

• One model, many contexts. The llama_model is loaded once and shared read-only. Each worker owns its own llama_context — contexts are not safe to share across threads for decode.
• Backpressure. The bounded queue offers push() (block until a slot frees) and try_push() (reject-when-full). The server uses blocking submit by default; try_submit() exposes the reject path for an HTTP front end.
• Shutdown. close() on the queue wakes all workers; each drains then exits; shutdown() joins them. No detached threads.

See ARCHITECTURE.md for the full picture.

Development approach

This project was built with AI assistance (Claude), with me acting as the engineer directing the work: setting the architecture, making the design decisions, and validating correctness at each step. I'm noting this openly because in 2026 the relevant question isn't whether AI generated code — it's whether the engineer can specify, evaluate, and own it.

What I brought to this specifically:

• Architecture and design decisions — the one-model / many-contexts model, the bounded-queue backpressure policy, the choice to build the serving layer over llama.cpp rather than reimplement a tensor engine, and where the concurrency boundaries sit.
• Correctness and debugging — diagnosing the mixed CPU/CUDA build failures (MSVC flags leaking onto the nvcc command line; the pinned-tag API mismatch), and reconciling the code against the llama.cpp C API.
• Benchmark design and interpretation — defining what to measure (TTFT, ITL, throughput under concurrency), running it on real hardware, and drawing the CPU-vs-GPU scaling conclusion that motivates the batching roadmap.
• Domain judgment — 25+ years in performance-critical, real-time, multithreaded C++ (surgical robotics, high-end film/video, algorithmic trading) is what let me specify this correctly and know when the output was right.

The compute kernel is llama.cpp; the serving layer, the systems design, and the engineering judgment are mine.

Roadmap

• OpenAI-compatible HTTP endpoint (/v1/chat/completions, SSE streaming)
• Continuous / in-flight batching (decode multiple sequences per step)
• Prefix / KV-cache reuse for a shared system prompt
• /metrics endpoint (Prometheus text format)

Terms

TTFT (time to first token), ITL (inter-token latency), continuous batching, KV-cache, quantization (GGUF: Q4_K_M / Q8_0 / F16).

License

This project's source code is released under the MIT License.