Open Weights, Closed Compute: The Privacy Hole in 'Open Source' AI APIs

Ask a privacy-conscious developer why they picked an open-weight model — Llama, Mistral, Qwen, DeepSeek — and you'll usually get some version of "because it's not sending my data to a corporation." Then ask where they're actually running it, and the honest answer is often: an API endpoint they don't own, operated by a company they've never audited, that logs every token the same way OpenAI or Anthropic would.

That's the gap this article is about. Not "open source doesn't mean private" in the abstract — that argument's been made. This is narrower and, for most people, more relevant: the privacy of an open-weight model depends entirely on who's running the compute, and "open weights" tells you nothing about that.

The Question Everyone Skips

When people evaluate an AI tool for privacy, they ask one question: is the model open source? That question feels like it settles things. It doesn't. It answers "can I audit the weights and the architecture," which is a real and valuable property. It does not answer "who sees my prompts, in plaintext, before or after inference runs."

Those are two separate axes:

Model openness — are the weights, training code, and architecture published and inspectable?
Compute custody — does the plaintext of your prompt ever leave hardware you control?

You can be maximally open on the first axis and zero on the second. That combination — open weights, someone else's servers — is exactly what a huge share of "privacy-focused" open-source AI usage actually looks like in practice, because running a 70B or 400B parameter model on a laptop isn't realistic for most people. So they reach for a hosted inference provider: OpenRouter, Together AI, Groq, Fireworks, Hugging Face's Inference Endpoints, Replicate. The model card says "open source." The privacy posture says otherwise.

What a Hosted Inference Provider Actually Sees

Walk through what happens when you send a prompt to one of these services. Your request hits their load balancer. It gets routed to a GPU node they operate, in a data center they lease. The full plaintext of your prompt sits in that node's memory for the duration of inference. Depending on the provider's logging configuration, it may also sit in a request log, a rate-limiting system, an abuse-detection pipeline, or a cache layer designed to speed up repeated queries — any of which can persist that plaintext well past the request itself.

None of this requires the provider to be acting in bad faith. Standard operational infrastructure — the stuff every API-based service runs for reliability, billing, and abuse prevention — needs to see your request in the clear to function. Rate limiters key off request content or hashes of it. Billing systems count tokens, which means parsing the payload. Abuse detection, almost by definition, has to look at what you sent. A provider doesn't need a business model built on your data to end up holding it; they need it just to keep the lights on.

Compare that to what "open source" implied you were signing up for. The pitch, implicitly, was: nobody's building a profile on me. But a profile doesn't require intent — it requires retention. And retention is the default state of any system that logs requests for debugging, which is nearly all of them.

The Spectrum Nobody Draws

Most privacy comparisons draw a two-point spectrum: closed AI (bad) versus open AI (good). The real spectrum has at least four points, and where you land matters more than which endpoint you started from:

Closed model, closed-provider hosted inference — GPT-4, Gemini, Claude via their native APIs. Weights are proprietary; compute is the vendor's.
Open model, third-party hosted inference — Llama or Mistral via OpenRouter, Together, Groq, etc. Weights are public; compute is still someone else's, run by a company whose logging practices you likely haven't read.
Open model, your own cloud instance — the same weights on a rented GPU box (Runpod, Lambda, a cloud VM) that only you access. Compute custody improves; you're still trusting the hosting provider's hypervisor and your own hardening.
Open model, your own hardware — Ollama or llama.cpp on a machine physically in your possession. This is the only point on the spectrum where "nobody but me sees the plaintext" is actually true by construction, not by policy.

The uncomfortable finding: point 2 is privacy-wise much closer to point 1 than to point 4. Swapping GPT-4 for Llama-3-70B via a hosted API changes your training-data exposure (your prompts probably aren't being harvested for the next foundation model) but does almost nothing for your operational exposure — the request still transits a third party's infrastructure in plaintext, gets logged by systems you don't control, and sits in an account tied to your API key, IP address, and billing details. You've changed which company can read your prompts. You haven't stopped a company from reading them.

A Concrete Walkthrough

Say you're building a side project that summarizes your own journal entries using Llama 3.1 70B, because you specifically didn't want to run that text through GPT-4. You don't have a GPU that can hold a 70B model, so you sign up for a hosted inference provider, generate an API key, and start sending requests.

Here's what actually happened to your journal entries, step by step:

Your client sends the raw text over HTTPS to the provider's API gateway. TLS protects it in transit — from everyone except the provider itself, which terminates the connection.
The gateway authenticates your API key and attaches your account ID, and typically your source IP, to the request for billing and abuse monitoring.
The request is routed to whichever GPU node currently has capacity. That node decrypts the request, holds your journal text in memory, and runs inference. Some providers batch multiple customers' requests onto the same GPU for efficiency, which means your plaintext shares process memory with other tenants' requests, however briefly.
The response is generated, sent back, and the request-response pair is very often written to a log — for debugging, for billing reconciliation, for abuse review — with a retention window set by the provider, not by you.
If the provider offers a prompt cache (common, since it cuts cost on repeated or similar requests), your journal text or a hash derived from it may be stored longer than the log retention window implies.

At no point in this sequence did the fact that Llama's weights are open source change any of it. The exact same five steps happen, nearly verbatim, if you swap Llama for GPT-4 and the hosted provider for OpenAI directly. The openness of the model bought you exactly one thing — the option to eventually run it yourself — and nothing about your current setup, unless you exercise that option.

What a Setup That Actually Delivers on the Promise Looks Like

If the goal was genuinely "nobody but me sees this," here's the setup that delivers it, roughly in order of how much it costs you in convenience:

Full local inference. Ollama or llama.cpp running a model sized to fit your own GPU or Apple Silicon RAM. Slower, more limited in model size, but the plaintext genuinely never leaves your machine. This is the only tier where the "open source AI = private AI" claim is literally true.
A cloud GPU instance you provision and tear down yourself. Rent a GPU-hour on Runpod or Lambda, load the model, run your batch job, destroy the instance. You're trusting the cloud provider's hypervisor isolation, but you're not sending requests to a shared multi-tenant inference API with its own logging stack layered on top.
A hosted inference provider with a verified zero-retention agreement. Some providers offer this for enterprise customers, in writing, sometimes backed by a compliance audit. It's not the default tier, and it's worth confirming in writing rather than assuming from the marketing copy.
A hosted inference provider on default terms. This is where most individual developers actually land, and it's the tier that gets misdescribed as "private" because the model happens to be open source.

None of this means the first tier is the only acceptable one. Plenty of use cases don't involve sensitive data and the convenience of hosted inference is a fair trade. The point is knowing which tier you're actually in, instead of assuming the model license moved you further up the list than it did.

Why This Trips Up Careful People Specifically

This isn't a mistake careless users make — it's a mistake careful users make, because the due diligence stops one step too early. Someone doing real privacy research will check: is the model's license permissive? Were the training data sources disclosed? Is there a community audit of the weights for backdoors? All good questions. Then they pick a hosted API to actually run inference and stop asking questions, because the hard part — evaluating the model — feels done.

But the model was never the part that could see your data. The inference provider is the part that can see your data, and it gets none of the scrutiny the model got. Nobody reads OpenRouter's or Together's data retention policy with the same energy they brought to comparing Llama's license against Mistral's. That asymmetry is the actual hole.

What Changes If You Take This Seriously

You don't need to self-host everything to close most of the gap. A few concrete moves, in order of effort:

Read the inference provider's data retention policy, specifically. Not their marketing page — their actual terms on log retention, whether prompts are used for anything beyond serving the request, and how long request logs are kept. Some hosted providers (a shrinking but real minority) offer zero-retention modes or enterprise agreements that disable logging. Most don't, by default.

Separate "which model" from "who's hosting it." If a project's privacy claim rests entirely on "we use an open-source model," ask the follow-up: hosted where, by whom, logged how? A vendor that can't answer that isn't further along than a closed-model vendor — they're just more comfortable, because "open source" sounds safe.

Reduce what a hosted provider can correlate, even if you keep using one. Routing API traffic through a VPN decouples your requests from your home or office IP address, so a provider logging plaintext prompts at least can't trivially tie that log to your physical location or ISP account. Proton VPN is a reasonable default here — no-logs policy backed by an external audit, based in Switzerland's stronger privacy jurisdiction, and it's one toggle rather than a new workflow. It doesn't stop the provider from seeing your prompt. It stops the prompt from being trivially linkable to you specifically.

Affiliate Disclosure: This article may contain affiliate links. If you make a purchase through these links, we may earn a small commission at no extra cost to you. We only recommend products we genuinely believe in. This helps support our work and allows us to continue providing free content.

If you're going to use a hosted, non-self-run AI product anyway, prefer ones that are upfront about their data practices over ones that are vague. Not every hosted AI product needs to be self-hosted to be a reasonable choice — but transparency should be a selection criterion, not an afterthought. Perplexity, for instance, publishes clear documentation on what it does with query data, which is a meaningfully different posture than a hosted-inference reseller with a two-paragraph privacy policy last updated in 2023. Reading the policy before adopting the tool, not after, is the actual habit worth building.

The Fox Takeaway

The question "is the model open source" answers a real question, just not the one people think it answers. It tells you whether the weights are auditable. It tells you nothing about whether the compute is yours. If you want an honest privacy audit of your AI stack, stop asking "which model am I using" and start asking "whose GPU is my plaintext sitting on right now, and what does their logging setup keep." For most hosted-inference setups running "open" models, the honest answer is: closer to a black box than the open-source framing suggests.

That's not an argument against hosted inference — sometimes it's the only realistic option. It's an argument for auditing the actual point where your data has custody, instead of stopping at the point where your due diligence feels satisfying.

Want the next one delivered straight to your inbox? Subscribe below and get practical AI privacy breakdowns — no hype, no fear-mongering, just what's actually happening under the hood.

Subscribe to PrivateAI →

Last updated: 2026-07-07