Architecture

Overview

                         submit(prompt)                     future<Result>
   caller / bench  ──────────────────────►  InferenceServer  ──────────────►  caller
                                                   │
                                                   ▼
                                       ┌────────────────────────┐
                                       │   BoundedQueue<Job>    │  capacity N
                                       │  (mutex + 2 cond vars) │  backpressure
                                       └────────────────────────┘
                                          │        │        │
                             pop()        │        │        │
                          ┌───────────────┘        │        └───────────────┐
                          ▼                        ▼                        ▼
                    ┌───────────┐            ┌───────────┐            ┌───────────┐
                    │ worker 0  │            │ worker 1  │    ...     │ worker M  │
                    │ own ctx   │            │ own ctx   │            │ own ctx   │
                    └───────────┘            └───────────┘            └───────────┘
                          │                        │                        │
                          └────────────┬───────────┴────────────────────────┘
                                       ▼
                          shared read-only  llama_model
                                       │
                                       ▼
                               llama.cpp / ggml
                             (compute kernel only)

Build variants

llama.cpp is pulled in via CMake FetchContent (pinned by GIT_TAG) and provides the entire compute backend — the box at the bottom of the diagram. It builds in one of two forms, selected at configure time:

CPU (default): ggml's CPU backend (AVX2 on this hardware).
CUDA (-DGGML_CUDA=ON): ggml's CUDA backend, offloading tensor math to the GPU. This flag flips only the compute kernel; nothing else in the tree changes.

Crucially, everything above that box is backend-agnostic and byte-for-byte identical across both builds — the bounded queue, the worker pool, one llama_context per worker, cooperative shutdown, and the metrics. The serving layer neither knows nor cares whether a decode step runs on CPU or GPU; it only sees llama_decode. The two builds live in separate directories (build/, build-cuda/) and can coexist.

This separation is what makes the benchmark comparison meaningful — the only variable between the CPU and GPU numbers is the kernel. And the results expose a scaling asymmetry that lives entirely in the backend, not the serving layer:

CPU: worker contexts can be handed disjoint physical cores, so concurrency raises aggregate throughput (modestly, until the memory bus saturates) and the serving layer's job is to keep all cores fed.
GPU: all worker contexts share one device. Parallel contexts overlap only until the GPU is saturated (here, by ~2 workers), after which more workers add no throughput and only latency + VRAM. Breaking that ceiling requires decoding multiple sequences in a single GPU step — continuous / in-flight batching — rather than N independent contexts. That is the motivation for the batching work on the roadmap, and it is a change to how the backend is driven, not to the queue/worker-pool design.

Threading model

Producer(s): whoever calls submit() / try_submit(). Thread-safe.
Queue: a single BoundedQueue<Job> guarded by one mutex and two condition variables (not_empty, not_full). This is the same bounded-buffer pattern used for the frame queue in QtOpenGLQuadPlayers, applied to inference requests.
Consumers: M worker threads. Each constructs its own llama_context (and sampler) on its own thread and loops: pop() a job, generate, fulfill the job's std::promise.

Why one context per worker

llama_model is immutable after load and is shared by reference across all workers. llama_context holds the KV-cache and decode state and is not safe to use from multiple threads concurrently, so each worker owns exactly one. Worker count is therefore the real concurrency knob (bounded by memory: each context allocates its own KV-cache).

Backpressure

The queue is bounded. Two policies:

push() blocks the producer until a slot frees (used by submit()).
try_push() returns false immediately when full (used by try_submit()), which is the correct behavior for an HTTP front end that should return a "server busy" status rather than buffer unbounded work.

Shutdown

shutdown() is idempotent (guarded by an atomic). It close()s the queue, which wakes every blocked worker. Each worker drains any remaining queued jobs, sees the closed-and-empty condition, and returns; shutdown() then joins them. No thread is detached and no in-flight request is dropped silently.

Instrumentation

Each RequestMetrics records four timestamps — enqueue, start (worker pickup), first token, end — plus prompt/generated token counts. From these:

queue wait = start − enqueue
TTFT = first token − start
ITL mean = (end − first token) / (generated − 1)
throughput = generated tokens / (end − first token)

AggregateStats summarizes a run: aggregate throughput over wall-clock time, TTFT p50/p95, mean ITL, and queue-wait p50.

Module map

File	Responsibility
`include/bounded_queue.hpp`	Generic bounded thread-safe queue + backpressure
`include/metrics.hpp`	Per-request timing + aggregate stats (header-only)
`include/inference.hpp` / `src/inference.cpp`	llama.cpp wrapper (the only TU that includes `llama.h`)
`include/server.hpp` / `src/server.cpp`	Worker pool, job dispatch, lifecycle
`src/main.cpp`	CLI, single-stream mode, concurrent benchmark mode

FilesExpand file tree

ARCHITECTURE.md

Latest commit

History