RAG-Based Technical Documentation Assistant

A FastAPI + LangGraph project for answering questions over technical documentation using Retrieval-Augmented Generation.

The assistant can ingest documentation from URLs or uploaded files, chunk and embed the content into ChromaDB, retrieve relevant chunks, grade retrieved documents, generate cited answers, and run a hallucination check before returning the final response.

Features

• URL-based documentation ingestion
• File-based ingestion for Markdown, text, and HTML files
• ChromaDB persistent vector store
• Configurable embeddings:
  • hash for smoke tests and local demos
  • sentence_transformers for local semantic embeddings
  • openai for OpenAI embeddings
  • gemini for Gemini API embeddings
• Configurable LLM provider:
  • stub for no-key local smoke tests
  • openai
  • groq
  • gemini
• LangGraph StateGraph workflow
• Query analysis and query rewriting
• LLM-based document grading
• Retry path when retrieval fails
• Answer generation with citations
• Hallucination/groundedness check node
• Cautious fallback when the answer cannot be verified
• Feedback endpoint

Architecture

flowchart TD
    A["POST /query"] --> B["Query Analysis"]
    B --> C["Retrieval from ChromaDB"]
    C --> D["Document Grading"]
    D --> E{"Relevant docs?"}
    E -- "Yes" --> F["Answer Generation"]
    E -- "No" --> G{"Retrieval retry left?"}
    G -- "Yes" --> H["Query Rewrite"]
    H --> C
    G -- "No" --> I["Insufficient Context Response"]
    F --> J["Hallucination Check"]
    J --> K{"Answer supported?"}
    K -- "Yes" --> L["Final Response"]
    K -- "No" --> M{"Generation retry left?"}
    M -- "Yes" --> N["Regenerate with stricter grounding"]
    N --> J
    M -- "No" --> O["Cautious Partial Response"]

Loading

Project Structure

.
├── app/
│   ├── main.py
│   ├── api/schemas.py
│   ├── core/config.py
│   ├── graph/
│   │   ├── nodes.py
│   │   ├── routing.py
│   │   ├── state.py
│   │   └── workflow.py
│   ├── ingestion/
│   │   ├── ingest.py
│   │   ├── loader.py
│   │   └── splitter.py
│   ├── llm/client.py
│   ├── prompts/templates.py
│   ├── retrieval/
│   │   ├── embeddings.py
│   │   └── vector_store.py
│   └── storage/jsonl.py
├── scripts/ingest_docs.py
├── data/raw/sample_urls.txt
├── tests/
├── requirements.txt
├── .env.example
└── README.md

Workflow Details

1. Ingestion

The ingestion pipeline accepts documentation URLs or uploaded files.

For each document, the app:

1. Fetches the URL or reads the file.
2. Extracts clean text from HTML when needed.
3. Preserves headings, paragraphs, lists, and code blocks.
4. Splits content into overlapping chunks.
5. Generates embeddings.
6. Stores chunks and metadata in ChromaDB.

Chunk metadata includes:

{
  "source": "https://fastapi.tiangolo.com/tutorial/body/",
  "title": "Request Body - FastAPI",
  "section": "Request Body",
  "url": "https://fastapi.tiangolo.com/tutorial/body/",
  "chunk_id": "https://fastapi.tiangolo.com/tutorial/body/#3"
}

2. Query Analysis

The query analysis node rewrites the user question into a retrieval-friendly query and optionally classifies it as:

• conceptual
• how_to
• troubleshooting
• api_reference
• unknown

3. Retrieval

The retrieval node embeds the rewritten query and performs top-k semantic search against ChromaDB.

4. Document Grading

Each retrieved chunk is graded by the LLM as:

• relevant
• irrelevant

Irrelevant chunks are filtered out. If no relevant chunks remain, the graph routes to query rewriting until the retry limit is reached.

5. Answer Generation

The generation node answers only from the relevant chunks and includes bracketed citations such as [1] and [2].

6. Hallucination Check

The hallucination check node compares the generated answer against the retrieved context and returns:

• supported
• partially_supported
• unsupported

If the answer is not supported, the graph regenerates once with stricter grounding. If it still cannot verify the response, it returns a cautious partial response.

Setup

1. Create a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

2. Install dependencies

pip install -r requirements.txt

3. Create environment file

Copy .env.example to .env.

Windows PowerShell:

Copy-Item .env.example .env

Recommended Configuration

For quick smoke testing without API keys:

LLM_PROVIDER=stub
EMBEDDING_PROVIDER=hash

For a stronger assignment demo with OpenAI:

LLM_PROVIDER=openai
LLM_MODEL=gpt-4o-mini
OPENAI_API_KEY=your_openai_key

EMBEDDING_PROVIDER=openai
OPENAI_EMBEDDING_MODEL=text-embedding-3-small

For a Gemini-based assignment demo:

LLM_PROVIDER=gemini
LLM_MODEL=gemini-2.5-flash-lite
GEMINI_API_KEY=your_gemini_key

EMBEDDING_PROVIDER=gemini
GEMINI_EMBEDDING_MODEL=gemini-embedding-2
GEMINI_EMBEDDING_DIMENSION=768
CHROMA_COLLECTION=technical_docs_gemini

Gemini Embedding 2 supports flexible output dimensions. This project uses 768 by default because it keeps local vector storage compact and is one of the recommended dimensions in the Gemini embeddings documentation.

For Groq chat generation with local embeddings:

LLM_PROVIDER=groq
LLM_MODEL=llama-3.1-8b-instant
GROQ_API_KEY=your_groq_key

EMBEDDING_PROVIDER=sentence_transformers
EMBEDDING_MODEL=all-MiniLM-L6-v2

Then install the optional embedding dependency:

pip install -r requirements-optional.txt

The stub and hash providers are included so the app can run locally without paid keys. For final submission, use a real LLM provider because document grading, generation, and hallucination checking are intended to be LLM-backed.

If you switch embedding providers or embedding dimensions, re-ingest the documents. Existing vectors from another embedding model are not compatible with Gemini vectors. The sample .env uses a separate technical_docs_gemini Chroma collection for this reason.

Run the App

uvicorn app.main:app --reload

Open:

• API docs: http://127.0.0.1:8000/docs
• Health check: http://127.0.0.1:8000/health

Ingest Documentation

Option 1: API request

curl -X POST "http://127.0.0.1:8000/ingest" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://fastapi.tiangolo.com/tutorial/body/",
      "https://fastapi.tiangolo.com/tutorial/dependencies/",
      "https://fastapi.tiangolo.com/tutorial/path-params/"
    ]
  }'

Option 2: Script

python scripts/ingest_docs.py ^
  https://fastapi.tiangolo.com/tutorial/body/ ^
  https://fastapi.tiangolo.com/tutorial/dependencies/ ^
  https://fastapi.tiangolo.com/tutorial/path-params/

On macOS/Linux, use backslashes instead of carets for line continuation.

Ask a Question

curl -X POST "http://127.0.0.1:8000/query" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "How does FastAPI validate request body data?"
  }'

Example response shape:

{
  "question": "How does FastAPI validate request body data?",
  "answer": "FastAPI validates request body data using Pydantic models... [1]",
  "sources": [
    {
      "source": "https://fastapi.tiangolo.com/tutorial/body/",
      "title": "Request Body - FastAPI",
      "section": "Request Body",
      "url": "https://fastapi.tiangolo.com/tutorial/body/",
      "chunk_id": "https://fastapi.tiangolo.com/tutorial/body/#0"
    }
  ],
  "metadata": {
    "query": "FastAPI request body validation with Pydantic models",
    "query_type": "how_to",
    "retrieval_attempts": 1,
    "generation_attempts": 1,
    "hallucination_check": "supported",
    "status": "answered"
  }
}

List Indexed Documents

curl "http://127.0.0.1:8000/documents"

Submit Feedback

curl -X POST "http://127.0.0.1:8000/feedback" \
  -H "Content-Type: application/json" \
  -d '{
    "question": "How does FastAPI validate request body data?",
    "answer_id": "optional-id",
    "rating": "thumbs_up",
    "comment": "Clear and useful."
  }'

Feedback is written to data/feedback.jsonl.

Run Tests

pytest

Design Decisions and Tradeoffs

• LangGraph as the orchestration layer: The workflow is explicit, inspectable, and easy to extend with additional self-correction nodes.
• ChromaDB for vector storage: Chroma is simple to run locally and stores metadata alongside chunks.
• Heading-aware chunking: The splitter tries to preserve section context and uses overlap so technical explanations and code examples are less likely to be separated.
• LLM document grading: Retrieval similarity alone can surface semantically close but unusable chunks. The grading node filters these before generation.
• Retry-limited self-correction: Query rewriting helps recover from poor retrieval, while retry limits prevent infinite loops.
• Hallucination checking: A separate node verifies the generated answer against retrieved context before returning it.
• Stub providers: Useful for development and smoke tests, but not a replacement for a real LLM-backed submission.

Improvements With More Time

• Add session-aware conversation memory for follow-up questions.
• Add web-search fallback through Tavily, Serper, or Exa.
• Add reranking with a cross-encoder.
• Add streaming responses.
• Add LangSmith tracing.
• Add a small Streamlit or Gradio UI.
• Store feedback in SQLite or Postgres instead of JSONL.

Assumptions

• The main query path answers from indexed documents, not live web browsing.
• URLs are fetched during ingestion and stored in the vector database.
• The user must ingest documents before asking questions.
• The hallucination checker is LLM-based when a real provider is configured.