Build a Private Document Q&A System With Ollama, LangChain, and ChromaDB
Last updated: 2026-06-15
Your documents never leave your machine. That's the whole point.
Cloud-based Q&A tools like ChatGPT file uploads or Google NotebookLM are genuinely impressive — but every PDF you send them is ingested by a third-party server, potentially used for training, and subject to data-breach risk. For contracts, medical records, financial statements, or proprietary research, that's an unacceptable trade-off.
The stack in this guide — Ollama + LangChain + ChromaDB — gives you the same "ask anything about your documents" capability with zero data exfiltration. Everything runs locally: the language model, the embedding model, and the vector index.
By the end you'll have a Python script that loads any folder of PDFs, indexes them, and answers natural-language questions with citations. No API keys. No accounts. No cloud.
What You're Actually Building (RAG in 60 Seconds)
Retrieval-Augmented Generation (RAG) works in two phases:
Indexing (runs once): Your documents are split into overlapping text chunks. Each chunk is converted into a vector embedding — a list of numbers that encodes semantic meaning. Those vectors are stored in a local database (ChromaDB).
Querying (runs every time): Your question is also converted to a vector. The system finds the chunks whose vectors are closest to your question vector, then hands those chunks to the language model as context. The model synthesizes an answer from what it was shown — not from general training data.
This matters for privacy because at no point does a chunk ever leave your machine. The LLM itself runs locally via Ollama.
Prerequisites and Hardware Reality Check
Before installing anything, be honest about your hardware:
| RAM | Usable models | Speed |
|-----|--------------|-------|
| 8 GB | llama3.2:1b, phi3:mini | Slow (2-4 tok/s) |
| 16 GB | llama3.2:3b, mistral:7b | Reasonable (5-15 tok/s) |
| 32 GB | llama3.1:8b, gemma2:9b | Comfortable |
| 64 GB+ | llama3.1:70b (quantized) | Fast |
For a dedicated local AI machine, a compact desktop with 32–64 GB RAM is the right starting point. The Minisforum UM790 Pro (AMD Ryzen 9 7940HS, upgradeable to 64 GB DDR5) runs 8B parameter models comfortably and fits behind a monitor.
Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.
Start Asking Your Documents Questions
You now have a fully local RAG system: Ollama serving the LLM and embeddings, LangChain wiring the retrieval chain, ChromaDB storing your vectors. Drop any PDF into docs/, run ingest.py, and start asking questions.
No API key. No subscription. No data leaving the room.
If you want to go deeper on local AI privacy — model isolation, disk encryption for your vector store, running this inside Docker — subscribe below and we'll send the advanced guide the week it publishes.
[Get the Advanced Local AI Privacy Guide →] (email capture)
Have a question about your specific document type or hardware setup? Drop it in the comments.