Skip to content

Repository files navigation

fizzlabai

fizzlabai system architecture

A solo-built deep research agent. You ask a question; the system runs a deterministic 8-phase pipeline over the open web and your own prior facts, and produces a cited Markdown answer with a live trace of every LLM call, tool call, and source URL it touched.

Built for architectural legibility and demo polish, not multi-tenant throughput. Optimized for the question shape "what's the current evidence on X?" — primary-source biased, recency-aware, with grounding, answer-fidelity, and depth-gap signals scored separately.

Status: V1 in development on a single Contabo VPS. Architecture reference: ARCHITECTURE.md (system-design source of truth). Canonical plan + ADR history: docs/final_plan.md. Phase-by-phase build journals: learning/.

The 8 phases

plan → search → read → synth → verify → (re_plan?) → polish → close
Phase Job Model tier
plan Decompose the question into 1–12 sub-queries with rationales + must_cover constraints. Flash
search Fan out web searches (Tavily or Serper); emit ranked candidate URLs tagged by sub-query. Flash-Lite
read Extract content from candidates in parallel via web_extract / http_fetch. Snippet-shortcut fallback when the LLM reader runs out of budget. Flash-Lite
synth Draft the answer with inline [citation] chips, grounded in the source corpus. Streams tokens to the UI. Flash
verify LLM judge: scores grounding_confidence, answer_fidelity, and depth_gaps independently; the routing layer reads all three (plus best-pass tracking) to pick the next move. Flash
re_plan (Conditional) On low confidence OR remaining depth gaps, emit gap-targeted sub-queries — grounding gaps target primary sources, fidelity gaps re-target on the dropped constraint, depth gaps use the verifier's targeted_query verbatim with expected_depth escalated. Flash
polish Rewrite the synth draft into the user-facing Markdown answer; token-streamed. Pre-polish coherence check + best-pass rollback (ship the highest-scoring pass, not the latest). Flash
close Persist citations + sources, then async-extract durable facts into the user's memory. User-pick

Routing decisions and failure-mode classifiers live in src/fizzlabai/orchestrator/graph.py. See docs/phases/ for per-phase prompts + rationale.

Stack

  • Runtime: Python 3.12, FastAPI, Pydantic v2, asyncio
  • Agent harness: deepagents 0.6.x — LangChain agent loop + LangGraph state graph, wrapped in a thin deterministic shell. Every real phase goes through agents/factory.py:create_phase_agent so the standard middleware stack (EventCapture, Budget, PhaseGuardrail) is structurally enforced.
  • Models: Google Gemini 3 Flash + Flash-Lite, per-phase routing. Claude Haiku + Gemini Pro kept as replay-only fallbacks. BYOK supported — users can plug their own Google key via /settings; operator key is the fallback.
  • Provider dispatch: One LLM-facing web_search / web_extract tool surface dispatches to either Tavily (default) or Serper / Google CSE, picked per user setting. The LLM is blind to which provider runs.
  • Reader: direct httpx with SSRF guards for static HTTP fetch + Tavily/Serper extract API for richer pages. A separate gVisor-isolated browser-use worker pod is the V2 design (CLAUDE.md rule #13) — current production runs in-pod.
  • Memory + storage: Postgres (app + LangGraph checkpointer), pgvector for dense embeddings, Tantivy BM25 for the filesystem-backed user memory, JSONL as canonical event log (Postgres + LangGraph are projections).
  • Observability: OpenTelemetry → Tempo / Loki / Prometheus / Grafana — always on. A custom LangChainOTelCallback bridges deepagents/LangChain events into the trace tree at the agent.ainvoke() call site. MLflow (opt-in via MLFLOW_TRACKING_URI) coexists on a separate LangChain callback for per-graph trace search + token/cost dashboards + session view via thread_id. See docs/mlflow-tracing.md.
  • Frontend: Next.js 15 (App Router), Tailwind, lucide icons. SSE for live phase events + token streaming.
  • Deploy: k3s on Contabo VPS, ArgoCD GitOps, cert-manager + Cloudflare DNS-01.

Quickstart

# clone + install
git clone <repo-url> fizzlabai && cd fizzlabai
uv sync

# bring up Postgres + observability stack
make obs-up

# apply DB migrations (one-time)
uv run alembic upgrade head

# frontend deps (one-time)
cd web && pnpm install && cd ..

# two terminals:
make api     # FastAPI on :8080
make web     # Next.js on :3000

Open http://localhost:3000, sign in via magic link, ask anything. If RESEND_API_KEY is empty the magic-link URL prints to the API server's stdout — good enough for solo dev.

First-run .env

Drop a .env at the repo root with at minimum:

# Models — pipeline is Gemini-only; operator key OR user BYOK at /settings.
GOOGLE_API_KEY=AIza...
# Optional but recommended for replay paths + tests:
ANTHROPIC_API_KEY=sk-ant-...

# Search providers — Tavily is the default; Serper is the user-pick alternative.
TAVILY_API_KEY=tvly-...
SERPER_API_KEY=...                    # optional

# Storage + auth
DATABASE_URL=postgresql+asyncpg://postgres:fizz_dev_only@localhost:5432/fizzlabai
JWT_SIGNING_KEY=<32+ char random — `python -c "import secrets; print(secrets.token_urlsafe(32))"`>
SECRETS_ENCRYPTION_KEY=<Fernet key — `python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"`>
BASE_URL=http://localhost:3000        # post-magic-link redirect target
KILL_SWITCH_TRIPPED=false

# Optional — magic-link delivery
RESEND_API_KEY=re_...                 # leave empty to log links to stdout
RESEND_FROM_ADDRESS=onboarding@resend.dev

# Optional — MLflow trace viewer (run `make mlflow` to start the server)
MLFLOW_TRACKING_URI=                  # empty = disabled; set to http://localhost:5050 to enable
MLFLOW_EXPERIMENT_NAME=fizzlabai

Full walkthrough: docs/local-dev.md.

Common commands

make obs-up       # Postgres + Tempo + Loki + Prometheus + Grafana via docker compose
make obs-down     # tear it all down
make api          # uvicorn on :8080
make web          # next dev on :3000
make mlflow       # MLflow tracking server on :5050 (opt-in trace viewer)
make eval         # run the golden suite end-to-end (live LLM, ~10-30 min)
make bench        # goldens + per-phase token/cost breakdown
make bench LABEL=before     # save the run for later diffing
make bench-compare A=before B=after
make gate         # ruff + mypy + pytest
make urls         # print every local web UI's URL

Per-phase test scope:

uv run pytest tests/test_phase07_graph.py -v      # graph + routing + middleware stack
uv run pytest tests/test_phase_guardrail.py -v    # LLM-round caps + soft-degrade banners
uv run pytest tests/test_cost_optim.py -v         # deterministic verify fallback
uv run pytest tests/test_agent_profiles.py -v     # harness-profile contracts

Local web UIs

Observability primer (panels, queries, what each store contains): docs/observability.md.

Repository layout

src/fizzlabai/
  agents/             # phase factories + ChatModelFactory + harness profiles
  api/                # FastAPI app, routes, schemas, dependencies
  orchestrator/       # the deterministic shell graph + routing logic
  middleware/         # EventCapture, Budget, PhaseGuardrail, DynamicModel
  tools/              # web_search, web_extract, http_fetch, think_tool, etc.
  backends/           # search/extract dispatcher (Tavily / Serper)
  memory/             # episodic JSONL writer + facts persister
  retrieval/          # BM25 + dense + RRF fusion
  obs/                # OTel + LangChain callback bridge + structured logging
  llm/                # budget gate + cost estimation
  core/               # settings, contracts, errors
  db/                 # async engine + session helpers
web/
  app/                # Next.js app router (pages + layouts)
  components/         # chat, task, sidebar, settings UI
  lib/                # API client + SSE event streaming
docs/
  ARCHITECTURE.md                # ← system design source of truth
  06-system-architecture.svg     # hero diagram (rendered at the top of this README)
  final_plan.md                  # canonical plan + ADR history
  local-dev.md                   # full first-run walkthrough
  observability.md               # Grafana / Tempo / Loki / Prometheus primer
  mlflow-tracing.md              # MLflow opt-in tracing setup
  phases/                        # per-phase prompts + design notes
learning/             # phase-by-phase build journals
data/
  episodes/           # JSONL per-task event logs (canonical)
  bm25/memories/      # per-user filesystem-backed memory index
  goldens/            # eval golden questions
  benchmarks/         # `make bench` output snapshots

Design invariants (load-bearing)

From CLAUDE.md — these rules gate every PR:

  • Async-only Python. No sync I/O outside cli/ and alembic/.
  • No relative imports. Always from fizzlabai.x.y import Z.
  • Pydantic at every public boundary. Inputs/outputs are typed models, never dicts.
  • mypy strict passes. # type: ignore requires an inline reason.
  • No print(), no os.environ outside core/settings.py, no Optional[T] (use T | None).
  • Structured logs only. logger.info("event_name", **kwargs) — never f-string.
  • No live LLM calls in tests. Tests use the fake-model path.
  • One chat-model factory. All ChatAnthropic / ChatGoogleGenerativeAI construction goes through agents/factory.py:ChatModelFactory.
  • One phase-agent factory. All create_deep_agent invocations go through agents/factory.py:create_phase_agent.
  • JSONL is canonical. Postgres + LangGraph checkpointer are projections of the JSONL event log, not the other way around.

What this is NOT

  • Not multi-tenant — one owner + magic-link visitors with strict daily caps.
  • Not optimized for throughput. Optimized for legibility + replayability.
  • Not a thin wrapper over create_deep_agent. The deterministic shell, the per-phase model routing, and the JSONL-first event log are first-class.
  • Not a chat UI. Not Claude Code. Not Cursor.

License

MIT

About

A personal Deep Research Agent

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages