Last updated: 2026-06-15

Your documents never leave your machine. That's the whole point.

Cloud-based Q&A tools like ChatGPT file uploads or Google NotebookLM are genuinely impressive — but every PDF you send them is ingested by a third-party server, potentially used for training, and subject to data-breach risk. For contracts, medical records, financial statements, or proprietary research, that's an unacceptable trade-off.

The stack in this guide — Ollama + LangChain + ChromaDB — gives you the same "ask anything about your documents" capability with zero data exfiltration. Everything runs locally: the language model, the embedding model, and the vector index.

By the end you'll have a Python script that loads any folder of PDFs, indexes them, and answers natural-language questions with citations. No API keys. No accounts. No cloud.

What You're Actually Building (RAG in 60 Seconds)

Retrieval-Augmented Generation (RAG) works in two phases:

Indexing (runs once): Your documents are split into overlapping text chunks. Each chunk is converted into a vector embedding — a list of numbers that encodes semantic meaning. Those vectors are stored in a local database (ChromaDB).

Querying (runs every time): Your question is also converted to a vector. The system finds the chunks whose vectors are closest to your question vector, then hands those chunks to the language model as context. The model synthesizes an answer from what it was shown — not from general training data.

This matters for privacy because at no point does a chunk ever leave your machine. The LLM itself runs locally via Ollama.

Prerequisites and Hardware Reality Check

Before installing anything, be honest about your hardware:

| RAM | Usable models | Speed |

|-----|--------------|-------|

| 8 GB | llama3.2:1b, phi3:mini | Slow (2-4 tok/s) |

| 16 GB | llama3.2:3b, mistral:7b | Reasonable (5-15 tok/s) |

| 32 GB | llama3.1:8b, gemma2:9b | Comfortable |

| 64 GB+ | llama3.1:70b (quantized) | Fast |

For a dedicated local AI machine, a compact desktop with 32–64 GB RAM is the right starting point. The Minisforum UM790 Pro (AMD Ryzen 9 7940HS, upgradeable to 64 GB DDR5) runs 8B parameter models comfortably and fits behind a monitor.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

Software prerequisites:

```bash

Verify Python version

python3 --version # needs 3.10+

Install uv (fast package manager — optional but recommended)

curl -LsSf https://astral.sh/uv/install.sh | sh

```

Step 1 — Install Ollama and Pull Your Models

Ollama handles model downloads, GPU offloading, and serving a local OpenAI-compatible API endpoint.

```bash

macOS / Linux

curl -fsSL https://ollama.com/install.sh | sh

Windows: download installer from https://ollama.com

```

Once installed, pull two models: one for chat, one for embeddings:

```bash

Language model (pick one based on your RAM)

ollama pull llama3.2 # 3B — good default

ollama pull mistral # 7B — better reasoning

Embedding model (small, fast, stays loaded separately)

ollama pull nomic-embed-text

```

Verify both are running:

```bash

ollama list

NAME ID SIZE MODIFIED

llama3.2:latest ... 2.0 GB ...

nomic-embed-text:latest ... 274 MB ...

```

Ollama runs a server at http://localhost:11434 automatically. Keep that in mind — LangChain will talk to it there.

Step 2 — Set Up Your Python Project

```bash

mkdir private-rag && cd private-rag

python3 -m venv .venv

source .venv/bin/activate # Windows: .venv\Scripts\activate

pip install langchain langchain-community langchain-ollama \

chromadb pypdf sentence-transformers

```

Create your project structure:

```

private-rag/

├── docs/ ← drop your PDFs here

├── chroma_db/ ← vector index (auto-created)

├── ingest.py ← indexing script

└── query.py ← Q&A script

```

Step 3 — Index Your Documents

ingest.py loads PDFs, splits them into chunks, embeds each chunk, and stores the vectors in ChromaDB.

```python

ingest.py

from pathlib import Path

from langchain_community.document_loaders import PyPDFDirectoryLoader

from langchain.text_splitter import RecursiveCharacterTextSplitter

from langchain_ollama import OllamaEmbeddings

from langchain_community.vectorstores import Chroma

DOCS_DIR = "./docs"

CHROMA_DIR = "./chroma_db"

EMBED_MODEL = "nomic-embed-text"

def ingest():

print("Loading documents...")

loader = PyPDFDirectoryLoader(DOCS_DIR)

documents = loader.load()

print(f" Loaded {len(documents)} pages from {DOCS_DIR}")

splitter = RecursiveCharacterTextSplitter(

chunk_size=800,

chunk_overlap=150,

separators=["\n\n", "\n", ". ", " "],

)

chunks = splitter.split_documents(documents)

print(f" Split into {len(chunks)} chunks")

print("Embedding and storing (this takes a few minutes on first run)...")

embeddings = OllamaEmbeddings(model=EMBED_MODEL)

vectorstore = Chroma.from_documents(

documents=chunks,

embedding=embeddings,

persist_directory=CHROMA_DIR,

)

print(f" Stored {vectorstore._collection.count()} vectors in {CHROMA_DIR}")

print("Done.")

if __name__ == "__main__":

ingest()

```

Run it:

```bash

python ingest.py

Loading documents...

Loaded 47 pages from ./docs

Split into 183 chunks

Embedding and storing...

Stored 183 vectors in ./chroma_db

Done.

```

Chunk size tuning: 800 characters with 150 overlap works well for dense PDFs (contracts, research papers). For looser content (blog exports, notes), try 500/100. Too small = fragmented context. Too large = diluted relevance.

Step 4 — Build the Query Interface

query.py loads the existing ChromaDB index, retrieves relevant chunks for your question, and streams the LLM's response with source citations.

```python

query.py

from langchain_ollama import OllamaEmbeddings, ChatOllama

from langchain_community.vectorstores import Chroma

from langchain.chains import RetrievalQA

from langchain.prompts import PromptTemplate

CHROMA_DIR = "./chroma_db"

EMBED_MODEL = "nomic-embed-text"

CHAT_MODEL = "llama3.2" # swap to "mistral" for better reasoning

PROMPT_TEMPLATE = """You are a precise research assistant. Use ONLY the context

below to answer the question. If the answer isn't in the context, say

"I don't have enough information in the provided documents."

Context:

{context}

Question: {question}

Answer:"""

def build_chain():

embeddings = OllamaEmbeddings(model=EMBED_MODEL)

vectorstore = Chroma(

persist_directory=CHROMA_DIR,

embedding_function=embeddings,

)

retriever = vectorstore.as_retriever(

search_type="mmr", # maximal marginal relevance — reduces redundant chunks

search_kwargs={"k": 5}, # retrieve 5 chunks per query

)

llm = ChatOllama(model=CHAT_MODEL, temperature=0)

prompt = PromptTemplate(

template=PROMPT_TEMPLATE,

input_variables=["context", "question"],

)

return RetrievalQA.from_chain_type(

llm=llm,

chain_type="stuff",

retriever=retriever,

return_source_documents=True,

chain_type_kwargs={"prompt": prompt},

)

def main():

print(f"Loading index from {CHROMA_DIR}...")

chain = build_chain()

print("Ready. Type 'exit' to quit.\n")

while True:

question = input("Question: ").strip()

if question.lower() in ("exit", "quit"):

break

if not question:

continue

result = chain.invoke({"query": question})

print(f"\nAnswer:\n{result['result']}\n")

print("Sources:")

seen = set()

for doc in result["source_documents"]:

src = doc.metadata.get("source", "unknown")

page = doc.metadata.get("page", "?")

key = f"{src}:{page}"

if key not in seen:

print(f" • {src} (page {page})")

seen.add(key)

print()

if __name__ == "__main__":

main()

```

Run it:

```bash

python query.py

Loading index from ./chroma_db...

Ready. Type 'exit' to quit.

Question: What are the payment terms in the contractor agreement?

Answer:

According to the contractor agreement, payment is due net-30 from...

Sources:

• docs/contractor-agreement-2025.pdf (page 3)

```

Step 5 — Add New Documents Without Re-Indexing Everything

Rebuild the index only for new files by checking what's already in ChromaDB:

```python

In ingest.py — replace the Chroma.from_documents call with:

vectorstore = Chroma(

persist_directory=CHROMA_DIR,

embedding_function=embeddings,

)

Get already-indexed sources

existing = {

m["source"]

for m in vectorstore._collection.get(include=["metadatas"])["metadatas"]

}

new_chunks = [c for c in chunks if c.metadata["source"] not in existing]

if new_chunks:

vectorstore.add_documents(new_chunks)

print(f" Added {len(new_chunks)} new chunks")

else:

print(" No new documents found")

```

Troubleshooting the Common Failures

"Connection refused" on embedding: Ollama isn't running. Start it with ollama serve in a separate terminal, or enable it as a system service (sudo systemctl enable ollama on Linux).

Answers hallucinate facts not in your docs: Your prompt template may be too permissive. Strengthen it: add "Do not use any knowledge outside the provided context" explicitly.

Slow embeddings on first ingest: Normal. nomic-embed-text processes ~50 chunks/minute on CPU. For 500+ page document sets, run ingest overnight or swap in a GPU machine.

ChromaDB version conflicts: Pin versions: chromadb==0.5.23 langchain-community==0.3.x. The ecosystem moves fast and minor version mismatches cause import errors.

What's Private, What Isn't

This stack keeps your data local by design — but two failure modes can break that guarantee:

Model download is not private. Ollama pulls models from ollama.com by default. For air-gapped environments, download the GGUF files manually and import them with ollama create.

ChromaDB telemetry is on by default. Disable it:

```python

import chromadb

client = chromadb.PersistentClient(

path=CHROMA_DIR,

settings=chromadb.Settings(anonymized_telemetry=False),

)

```

With those two items handled, no document content, no query text, and no answer text touches an external server.

Take Your Private AI Further

The pipeline you built today handles single-user, local workloads well. The natural next step is putting a proper web UI in front of it — Open WebUI works with Ollama out of the box and turns your local RAG system into a shareable intranet tool your whole team can use without accounts.

Start Asking Your Documents Questions

You now have a fully local RAG system: Ollama serving the LLM and embeddings, LangChain wiring the retrieval chain, ChromaDB storing your vectors. Drop any PDF into docs/, run ingest.py, and start asking questions.

No API key. No subscription. No data leaving the room.

If you want to go deeper on local AI privacy — model isolation, disk encryption for your vector store, running this inside Docker — subscribe below and we'll send the advanced guide the week it publishes.

[Get the Advanced Local AI Privacy Guide →] (email capture)

Have a question about your specific document type or hardware setup? Drop it in the comments.