How to Run AI Locally: The Complete Guide to Private LLMs That Never Phone Home

Every prompt you send to ChatGPT, Claude, or Gemini is logged, potentially used for training, and stored on servers you have zero control over. For developers, legal professionals, security researchers, and anyone handling sensitive data, that's not a trade-off — it's a liability.

The good news: you don't have to choose between capable AI and actual privacy. Modern open-weight models running on consumer hardware are now competitive with GPT-4 for most day-to-day tasks. This guide walks you through a complete local AI stack — from installing your first model to securing everything that touches it.

Last updated: 2026-03-22

Why "Private Mode" Isn't Actually Private

Before we build anything, let's be precise about the threat model. Most cloud AI providers offer some version of "privacy mode" or claim not to train on your data. Read the fine print:

API usage often has separate data retention policies from consumer products
Even with training opted out, your prompts still transit their infrastructure and are subject to their security posture
Legal hold, government requests, and data breaches are outside your control
Enterprise agreements don't protect you if you're a solo developer or small team

Running locally eliminates all of this. Your prompts go: keyboard → RAM → GPU → response. No network hop. No log entry. No terms of service to re-read every six months.

What You Actually Need to Run AI Locally

The barrier is lower than most people think. Here's the honest hardware breakdown:

Minimum viable (8B models, ~8GB VRAM or unified memory):

Apple M2/M3/M4 with 16GB RAM — excellent performance via Metal
Any NVIDIA GPU with 8GB VRAM (RTX 3070, 4060, etc.)
AMD RX 7900 XT+ with ROCm support (Linux only, more setup)

Comfortable (13B-34B models, real GPT-3.5 competition):

Apple M3 Pro/Max with 36GB unified memory
NVIDIA 3090/4090 (24GB VRAM) — the classic local AI workhorse

Overkill / serious (70B+ models):

Mac Studio or Mac Pro with 64-192GB unified memory
Multi-GPU setups or consumer AI boxes

If you're on Apple Silicon, you already have one of the best local AI platforms available. Unified memory means the GPU and CPU share the same pool, and llama.cpp has outstanding Metal support.

Installing Ollama: The Easiest On-Ramp

Ollama is the de facto standard for running open-weight models locally. It handles model downloading, quantization selection, and serves a local API compatible with the OpenAI spec — which means most AI tools that talk to ChatGPT can be redirected to your machine with one config change.

macOS:

```bash

brew install ollama

ollama serve # starts the local API server on localhost:11434

```

Linux:

```bash

curl -fsSL https://ollama.com/install.sh | sh

```

Windows: Download the installer from ollama.com. WSL2 with CUDA passthrough also works if you prefer a Linux environment.

Once Ollama is running, pull your first model:

```bash

ollama pull llama3.2 # Meta's 3B model — fast, surprisingly capable

ollama pull qwen2.5:14b # Excellent reasoning, strong code

ollama pull deepseek-r1:14b # Best-in-class for technical reasoning

ollama pull mistral-small # Fast, good for summarization

```

Test it immediately:

```bash

ollama run llama3.2 "Explain OAuth2 in plain English"

```

Everything that just happened: local only, zero network egress after the initial model download.

Choosing the Right Model for Your Work

Not all open-weight models are equal. Here's a practical guide for common use cases:

Code generation and review: qwen2.5-coder:14b or deepseek-coder-v2. Both outperform GPT-3.5-turbo on benchmarks for completions and bug finding.

Document analysis and summarization: llama3.2:3b for speed, llama3.1:8b for quality. Feed these your contracts, specs, and internal docs without the data ever leaving your machine.

Reasoning and analysis: deepseek-r1 series. Shows its chain-of-thought reasoning — useful when you want to audit how the model reached a conclusion.

Multilingual work: qwen2.5:7b — strong across 30+ languages.

Instruction following and chat: mistral-small or gemma3:9b — efficient and well-tuned for conversation.

A note on quantization: Ollama defaults to Q4_K_M quantization, which cuts model file size roughly in half with minimal quality loss. If you have headroom, Q6_K or Q8_0 give noticeably sharper outputs for technical work.

Connecting Local AI to Your Existing Tools

This is where local LLMs become genuinely useful rather than just a demo. Because Ollama exposes an OpenAI-compatible API, you can drop it into most AI-enabled tools with a URL change.

VS Code / Cursor:

Configure the API base URL to http://localhost:11434/v1 and use any Ollama model name as the model ID. You now have GitHub Copilot-equivalent code assistance with zero telemetry.

Continue.dev (VS Code extension):

Purpose-built for local AI coding assistance. Native Ollama integration, no config gymnastics.

Open WebUI:

A polished ChatGPT-style interface that connects to Ollama. Run it in Docker:

```bash

docker run -d -p 3000:8080 \

--add-host=host.docker.internal:host-gateway \

-v open-webui:/app/backend/data \

ghcr.io/open-webui/open-webui:main

```

Navigate to localhost:3000 — you have a full-featured AI chat interface with conversation history, model switching, and file uploads, all running locally.

n8n / Make / Zapier: Use Ollama's REST API (POST localhost:11434/api/generate) as a custom HTTP action to add local AI to your automations.

Securing Everything Around Your Local AI Stack

Running models locally handles the most sensitive part — your prompts and responses. But there are adjacent attack surfaces worth hardening.

Model downloads: Always pull from ollama.com or Hugging Face with verified checksums. Malicious fine-tunes exist. Check that the model card comes from the original developer organization.

Network exposure: By default, Ollama only listens on 127.0.0.1. Keep it that way unless you explicitly need LAN access. If you're sharing a machine or running a home server, add auth middleware (nginx + htpasswd at minimum).

Prompt/output storage: If you're logging conversations or feeding Ollama outputs into other workflows, those files can contain sensitive data. Don't sync them unencrypted.

For encrypted cloud backup of AI outputs, notes, and anything sensitive your local stack produces, Proton Drive gives you end-to-end encrypted storage with desktop sync. Unlike Dropbox or Google Drive, Proton cannot read your files — the encryption keys never leave your device. Free tier covers 1GB; paid plans start at €3.99/mo with a generous referral structure.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

Full-disk encryption: Enable FileVault (macOS) or LUKS (Linux) on any machine running local AI if there's even a chance it contains sensitive model outputs or conversation logs. This is table stakes.

When Local AI Isn't Enough: Privacy-Forward Cloud Fallbacks

Local models are remarkable but they have real limits — 128K context windows are now common for cloud models, while even well-quantized local models on consumer hardware often cap at 8-32K tokens. For tasks involving huge codebases, lengthy documents, or complex multi-step reasoning, you occasionally need cloud capacity.

When you do go to the cloud, choose providers with explicit privacy commitments and minimal data retention. Perplexity AI has become a genuine alternative to Google Search for research tasks — it cites sources, doesn't build an ad profile around your queries, and the Pro plan offers $0 data training opt-out with API access. Particularly useful for privacy-conscious users who need up-to-date information that local models can't provide.

The workflow: default to local for anything sensitive, use cloud AI for truly demanding tasks with non-sensitive inputs, and never paste credentials, PII, or confidential code into any cloud interface regardless of their privacy policy.

Handling Sensitive Files: Encrypted Sync for Your AI Workspace

If you're using local AI for work — drafting documents, analyzing data, reviewing contracts — those output files need the same protection as your prompts. Standard cloud sync (iCloud, Dropbox, Google Drive) scans file contents for compliance and other purposes.

Tresorit is the encrypted alternative used by law firms and healthcare providers precisely because they never hold your decryption keys. Files are encrypted client-side before upload, so even a subpoena to Tresorit returns ciphertext. The Business plan includes secure link sharing, which matters if you're collaborating on AI-generated documents with clients or teammates.

The practical setup: point Ollama's output directory (or your Open WebUI conversation exports) to a Tresorit sync folder. Sensitive AI work is now backed up, versioned, and encrypted without any extra steps.

A Complete Private AI Workflow in Practice

Putting it all together, here's what a privacy-maximizing daily AI workflow looks like:

Ollama running locally — handles 90% of queries: code review, drafting, summarization, Q&A
Open WebUI — browser interface with conversation history, all stored in a local Docker volume
Proton Drive — encrypted backup of exported conversation logs and AI-generated docs
Tresorit — encrypted sync for the AI workspace folder shared with collaborators
Perplexity Pro — cloud fallback for research queries requiring current information, with training opt-out enabled

Total cost: Ollama (free), Open WebUI (free), Proton Drive free tier (free), Tresorit Business (~$14/user/mo), Perplexity Pro ($20/mo). Compare that to the privacy cost of feeding sensitive work product into ChatGPT or Gemini indefinitely.

The Bottom Line on Local AI Privacy

Running AI locally isn't about being paranoid — it's about being precise. You don't accept software licenses without reading them; you shouldn't accept AI privacy policies without understanding what they mean for your specific use case.

The stack outlined here — Ollama + an open-weight model on your own hardware — handles the core risk. Layering encrypted storage on top of it closes the surrounding gaps. What you're left with is an AI assistant that's genuinely yours: no vendor lock-in, no terms of service roulette, no wondering whether your client's contract language just became a training example.

Start with ollama pull llama3.2 tonight. The productivity difference from cloud AI is smaller than you think. The privacy difference is not.

Frequently Asked Questions

What does "running AI locally" actually mean?

Running AI locally means the AI model and all of its computations run on your own computer rather than on a company's cloud servers. When you type a prompt into a local AI setup like Ollama, the text never leaves your device — it is processed by software running on your CPU or GPU and the response is generated entirely on your machine. There is no network connection to any external server, no logging, and no entity other than your own hardware has access to your conversations.

Is locally-run AI actually more private than using ChatGPT or Claude?

Yes, categorically. Cloud AI services receive your prompts on their servers, where they are logged, potentially reviewed by employees, potentially used for training, and subject to legal requests and security breaches. With local AI, the only entity that ever sees your prompts is your own computer. There is no network transmission, no terms of service that can change, no breach that can expose your conversation history, and no vendor that can be subpoenaed for your data.

Do I need an expensive or powerful computer to run AI locally?

Not as powerful as most people assume. A modern laptop with 16 GB of RAM can run capable 7B to 8B parameter models that handle most everyday tasks well. Apple Silicon MacBooks are particularly efficient — the M2 MacBook Air runs local AI models faster than many gaming PCs costing three times as much. You sacrifice some raw capability on complex reasoning tasks with more modest hardware, but the full privacy benefit is present regardless of model size.

What are the main downsides of running AI on your own hardware?

The honest trade-offs: local models generally trail the frontier (GPT-4o, Claude Opus) in raw reasoning ability, particularly for complex analytical tasks. They cannot browse the internet or access real-time information. Initial setup requires more technical comfort than opening a browser tab. And the most capable local models require 32 GB or more of RAM to run at full quality. For most tasks — writing, coding, summarization, document analysis — the capability gap has narrowed significantly, but for cutting-edge research tasks, cloud models still hold an edge.

Get the Private AI Weekly

Local LLM releases, privacy tools, and setup guides — straight to your inbox. No tracking, no ads.