About — Document Chatbot

Motivation

Why build another document chatbot?

Most chatbot demos end at a prompt. The genuinely interesting engineering is earlier: deciding how a document becomes chunks, how those chunks become vectors, which passages are retrieved for a question, and how the answer proves where it came from. Those choices determine whether the system is useful or merely convincing.

This project treats RAG as a pipeline with inspectable stages. Ingestion, retrieval, generation and transport live in separate modules. The frontend is deliberately plain, the configuration is environment-driven, and the tests run against isolated temporary storage with fake models. The goal was a system that can be demonstrated, explained and debugged without hidden services.

The design stance is simple: answer from the provided context, show the passages, fail with a readable message, and never present an unverified capability as a feature. Chat generation uses an external API; embeddings and the vector library run locally.

Grounded does not mean guaranteed. The system is designed to make its evidence visible so a reader can check it.

Architecture

Two paths, one modular backend

Ingestion writes to the vector library. Question answering reads from it. Select any component to see what it owns.

1Client

2API and services

3Ingestion path

4Question-answering path

5Storage and configuration

Module map

Who owns what

The backend is grouped by responsibility rather than by layer alone, so each stage of the pipeline has one obvious home.

app.py

Application entrypoint

Creates the FastAPI app, configures CORS and structured error handlers, logs startup configuration, and mounts the static frontend at /app.

api/

Transport layer

Routes for chat, documents and health. Pydantic schemas validate requests and shape responses; api/errors.py builds the shared error envelope.

services/

Application services

document_service.py streams and validates uploads, lists the library and deletes documents. chat_service.py validates document selection, builds the retriever and runs the chain.

ingestion/

Load, split and identify

Format-specific loaders, the recursive character splitter, and the pipeline that computes the content hash and attaches doc_id, filename and chunk_index.

embeddings/

Local vectors

Loads sentence-transformers/all-MiniLM-L6-v2 once per process, on CPU, with normalized embeddings.

vectorstore/

Persistent storage

Owns the Chroma collection: upserting chunks, pruning stale IDs, listing documents, counting chunks and deleting by doc_id.

retrieval/

Similarity search

Builds a similarity retriever with a configurable k and an optional doc_id filter for scoped questions.

chains/

Prompt and answer chain

The real prompt template and the LCEL chain that retrieves, formats context, calls the model, parses text and short-circuits empty retrieval.

generation/

Model access and errors

Creates the cached Groq client and translates provider failures into typed exceptions the API can map to readable responses.

config/ + utils/

Configuration and shared helpers

Environment-driven settings, filename sanitisation, document-ID validation, file-signature checks and logging setup.

Engineering decisions

The trade-offs behind the code

These are the choices that shaped the implementation, with the reasoning that made them worth keeping.

ADR-1 Groq for chat, local embeddings

Chat generation uses Groq's openai/gpt-oss-20b at temperature 0. Embeddings use a local sentence-transformers model. This keeps ingestion free of a second API key and per-chunk cost, while keeping generation fast and inexpensive.

The consequence is a split dependency: retrieval works locally, but a final answer still requires network access to Groq and is subject to its free-tier limits.

ADR-2 LCEL instead of legacy chains

The answer path is composed with LangChain Expression Language: retrieve, format documents, build the prompt, call the model, parse the string. LCEL keeps the data flow explicit and makes the empty-retrieval branch easy to isolate.

ADR-3 Normalized embeddings, no exposed score

Vectors are normalized and Chroma uses its default L2 index. The API returns the retrieved passages and metadata, but not a similarity score. Exposing a raw distance without documenting the metric would invite comparisons that do not mean what they appear to mean.

ADR-4 Content-hash IDs and deterministic chunks

doc_id is the SHA-256 of the file bytes, and each chunk ID combines that hash with its index. Re-uploading identical content replaces the existing document instead of duplicating it.

Replacement upserts the new chunks first and only then prunes IDs that no longer exist, so a failed update does not delete the previous working version first.

ADR-5 Metadata travels through every stage

Every chunk carries doc_id, filename and chunk_index, plus page metadata supplied by the loader. That is what makes document scoping and page-level citations possible without a second index.

ADR-6 Persistent Chroma, one worker in development

Chroma runs in persistent local mode under data/chroma_db. The embedding model, vector store and LLM client are cached per process, so local development uses a single Uvicorn worker.

Scaling to multiple workers would mean each process loads its own model and opens its own client; that is a deployment concern this local project does not pretend to solve.

ADR-7 Skip the LLM when nothing is retrieved

If retrieval returns no chunks, the chain returns a fixed message explaining that no relevant information was found. Calling the model with empty context could only produce an ungrounded answer, and it would make an infrastructure failure harder to distinguish from a genuine miss.

ADR-8 One error envelope, readable in the UI

Application errors return a consistent JSON shape with a code and message. Provider failures such as rate limits and timeouts are translated before they reach the browser. The frontend renders these as a dismissible banner, inline feedback or a toast — never a browser alert.

ADR-9 Same-origin static frontend

FastAPI serves the frontend from the same process at /app, so there is no separate frontend server and no CORS requirement by default. The UI is plain HTML, CSS and JavaScript with no build step.

ADR-10 Isolated tests with fake models

The normal test suite uses temporary Chroma and upload directories, fake embeddings and a fake chat model. A real-provider test is opt-in through an environment variable. This keeps the suite fast, deterministic and independent of network access.

Key features

What the implementation actually provides

PDF, TXT and DOCX ingestion, plus a paste-text dialog that uses the same pipeline.
Local 384-dimension embeddings with a persistent Chroma vector library.
Grounded prompting with an explicit instruction to admit when the context does not contain the answer.
Source citations with filename, chunk index and page metadata when available.
Document-scoped retrieval and a configurable top-k between 1 and 20.
Content-addressed document IDs with safe replacement of re-uploaded files.
Readable, structured errors and a live health indicator in the UI.
Light and dark themes, keyboard-accessible controls and reduced-motion support.

Tech stack

Pinned tools, no hidden services

Chat inference is the only external call. Document processing, embeddings and vector storage run locally.

APIFastAPI 0.162Typed routes, validation, static hosting

LCLangChain 1.xLoaders, splitter, retriever, LCEL chain

CHChromaDB 1.5Persistent local vector collection

GQGroq + gpt-oss-20bChat generation, temperature 0

HFsentence-transformersall-MiniLM-L6-v2, 384-dim, CPU

PYpypdf + docx2txtPDF and Word text extraction

PDPydantic SettingsEnvironment-driven configuration

JSVanilla HTML/CSS/JSNo framework, no build step

Limitations and future work

What this project does not do

Being explicit about the boundary is part of the design. Nothing below is implemented today.

Known limitations

Retrieval is similarity search only. There is no hybrid keyword search and no reranking stage.
Answer generation requires the Groq API; free-tier rate and token limits can reject requests.
Scanned or image-only PDFs will not yield text without OCR, which is not implemented.
Complex DOCX tables and layouts may lose structure when extracted as plain text.
Chunking is character-based with fixed defaults, not semantic or layout-aware.
The app has no authentication or per-user isolation; it is a local, single-user tool.
Responses are not streamed; the answer appears when the request completes.
Grounded prompting reduces invented answers but cannot guarantee correctness.

Not implemented — future work

Possible next steps

Hybrid retrieval with a keyword index, followed by a reranker.
OCR for scanned documents and better table extraction.
Streaming answers with incremental source rendering.
An evaluation harness for retrieval quality and answer faithfulness.
Authentication and per-user document libraries.
Background ingestion for very large files and richer document metadata.
Deployment hardening: containers, worker strategy and an external vector store.

Credits

Built and maintained by

Om-codex

Design, backend, frontend and documentation. Built as a complete RAG engineering project with FastAPI, LangChain, ChromaDB, sentence-transformers and Groq.

GitHub profile Replay the engine tour Open Chatbot

Model and library credits belong to their respective projects: LangChain, Chroma, Hugging Face sentence-transformers, Groq and FastAPI.