Where Does Your Data Actually Go When You Use AI?

Every week a regulated-industry buyer asks me some version of the same question: if we use AI, where does our data actually go, and do we need a private or on-prem model to stay on the right side of the law? It is the correct question, and most vendors answer it badly. They wave a data-center map, point at a region inside your country, and call it compliant. When our team does AI consulting for banks, hospitals, and government suppliers, the first thing we untangle is the difference between two words vendors love to blur: residency and sovereignty.

Get that distinction right and the rest of the decision, public API, private cloud, or on-prem, falls out of it. Get it wrong and you either overspend on infrastructure you never needed, or you sign off on an architecture that quietly breaks the very rule you were trying to satisfy. So let me walk through where your data really goes, what the two words mean, and how to pick the tier that matches your actual obligation instead of the vendor's pitch.

Where your data goes when you call an AI API

The short answer: when you call a public AI API, your prompt and everything in it leaves your network, travels to the vendor's servers, and the model runs there, on infrastructure you do not own, in a location and under a legal jurisdiction the vendor sets, then the reply comes back. That round trip is the whole issue. For a marketing blurb it does not matter. For a patient record, a loan application, or a case file, it matters a great deal.

What happens to your text once it lands depends entirely on the tier you are on. Consumer chat products may log inputs and, on some plans, use them to improve models. Enterprise and developer API tiers usually offer contractual no-training with short or zero retention. Same model, very different data posture. The mistake I see most often is a team validating an idea by pasting real customer data into a consumer chatbot, then assuming the enterprise contract they signed months later somehow covers what already left the building.

Data residency vs data sovereignty (the trap)

Data residency is physical: which country your data is stored and processed in. Data sovereignty is legal: whose laws govern that data and who can compel access to it. They sound like the same thing. They are not, and the gap between them is where compliance projects quietly fail.

Here is the trap in one example. A US-headquartered cloud provider offers you a region inside India or the EU. Your data now physically sits in-country, so you have residency. But the US CLOUD Act, passed in 2018, lets US authorities compel a US-controlled provider to produce data it holds, wherever in the world that data lives. So the provider can lawfully be ordered to hand over data stored in Mumbai or Frankfurt. You have residency, you do not have sovereignty, and for regulated data that distinction is the whole ball game. This is not hypothetical lawyering. It is the exact concern that led the EU's top court to strike down the Privacy Shield transfer framework in the 2020 Schrems II ruling, and it is why more than 75 percent of Europe and Middle-East enterprises are expected to geopatriate workloads to sovereign clouds by 2030.

When you actually need a private or on-prem LLM

You need a private or on-prem LLM when your data carries a legal or contractual requirement a public API cannot meet, not merely because the data feels sensitive. That is the line. Everything sensitive does not need to be self-hosted, but a genuine mandate is non-negotiable. The triggers that actually force the strict tier:

• Localization mandates. India's approach is a mix. The DPDP Act generally permits cross-border transfer except to countries the government restricts, but sector rules bite harder. The RBI requires payment system data to be stored only in India, which is a hard localization requirement, not a preference.
• Sovereignty requirements. Data that legally cannot sit under foreign jurisdiction, common in government, defense, and critical-infrastructure work.
• Contractual control. A client contract that says their data stays in your custody and inside a named country.
• Trade secrets and privilege. Legal case files, M&A material, or core IP you will not place in a third party's hands at any price.

The counterweight matters just as much. Plenty of workloads that feel scary are perfectly fine on a private cloud deployment with the right contract and some de-identification. The EDPB has pointed to keeping data on infrastructure you control as among the strongest supplementary protections available, but strongest is not the same as always necessary. On-prem is the most powerful and most expensive lever you have. Pull it for a real mandate, not a vibe.

The three deployment tiers, compared

There are really only three places an LLM can run for you, and they trade control against cost and capability in a predictable way. Public API gives you the most capability for the least money and the least control. On-prem gives you the most control for the most money and a capability gap. Private VPC sits in between and is where most regulated work lands.

What each tier costs you, in money and capability

The real cost of moving toward control is two things: money and model capability, and they both move against you at once. A public API is cheapest because you pay per token and carry no infrastructure. A private VPC deployment adds compute, data-egress charges, and setup, but often runs the same frontier models through Azure OpenAI, Bedrock, or Vertex, so you keep capability while gaining residency control. On-prem is the most expensive by a distance, because you are buying GPUs and hiring the people to keep them healthy, and you accept open-weight models that trail the frontier by a shrinking but real margin.

The economics look a lot like any other cloud-versus-owned-infrastructure decision, and the same discipline we lay out in our cloud migration checklist applies here: watch egress, right-size the hardware, and do not pay for control you will never actually use. This is not a fringe concern to budget around either. Analysts put the sovereign-cloud market near 195 billion dollars in 2026, which tells you how many organizations are pricing exactly this trade right now.

A decision checklist for regulated data

Before you pick a tier, answer these in order. Most teams that get stuck skipped the first two and jumped straight to hardware.

• Name the actual rule. Is it RBI payment-data localization, the DPDP Act, GDPR, HIPAA, or a sector regulator? Point to the clause. Do not reason from "it feels sensitive so it must stay here."
• Separate residency from sovereignty. Does the rule require the data to sit in-country, to stay under domestic legal control, or both? That answer alone narrows your tier.
• Classify the data. Public, internal, confidential, or regulated PII, PHI, or financial. Most systems hold a mix, and only a slice usually needs the strict tier.
• Ask what can be de-identified. If you can tokenize or mask the regulated fields before the model ever sees them, a cheaper tier often becomes compliant.
• Read the contract, not the map. Confirm no-training, retention period, region pinning, and the sub-processor list in writing.
• Check who can compel access. Is the operator foreign-controlled, and does a law like the CLOUD Act reach it?
• Price the whole thing. Weigh breach and non-compliance cost against the true cost of running on-prem, including the team it takes to keep it running well.

What regulated and Indian buyers get wrong

The short answer: they confuse the physical for the legal, and they treat the deployment tier as the whole compliance story when it is only half of it. The specific errors I see most, again and again:

• Treating a local data center as full compliance. "It is in the Mumbai region" answers residency, not sovereignty, and regulators increasingly ask the second question.
• Assuming on-prem is automatically safer. A neglected on-prem box with weak access control and no patching is less secure than a well-run VPC. On-prem gives you control, not safety by default.
• Over-provisioning. Going full self-hosted for data that could have been tokenized and sent to an enterprise API, then carrying GPU and staffing costs you never needed.
• Ignoring contract terms already on offer. Enterprise no-training and zero-retention options exist and are badly underused.
• Letting shadow AI run. The biggest real leak is rarely the architecture, it is staff pasting regulated data into consumer chatbots, which is why a shadow AI governance playbook matters as much as the deployment tier you pick.

How we architect for data control

We start where the vendor pitch ends: with your data classification and the exact clause you are bound by, not a map of regions. Once we know which slice of data is genuinely regulated, the architecture usually designs itself. Most of the time the cheapest compliant answer is a tiered one, not an all-on-prem one, and building it that way is the same classification-first approach we bring to every custom AI development engagement.

As a CMMI Level 5 firm with more than 80 engineers delivering across the UK, US, UAE, Australia and India, we build both the private-cloud and the self-hosted end of this, and we tell you plainly when the expensive tier is not the one you need. If you are staring at a regulator's clause and a vendor's data-center map and cannot tell whether you need a pinned VPC or a truly sovereign build, tell us what data you handle and we will map the obligation to the cheapest architecture that actually satisfies it.

Frequently Asked Questions

Do I need a private or on-prem LLM for regulated data?

Not always. The test is what your regulator actually requires, residency (where data sits) or sovereignty (who controls it). If tokenizing sensitive fields plus a private VPC deployment with no-training, region-pinned contract terms satisfies the rule, that is usually enough. Reserve on-prem for hard mandates like payment-data localization or data that legally cannot touch foreign-controlled infrastructure. Classify first, then choose.

What is the difference between data residency and data sovereignty?

Residency is physical, which country your data is stored and processed in. Sovereignty is legal, whose laws govern it and who can compel access. A US provider's Mumbai region gives you residency, but under the US CLOUD Act that provider can still be ordered to produce the data, so you do not have sovereignty. The two often conflict.

Does an "India data center" or "EU region" make me compliant?

It helps with residency, but on its own it does not guarantee compliance. If the operator is a foreign-controlled company, your data can still fall under that country's disclosure laws no matter where the servers sit. Check who legally controls the operator, the contract terms on training and retention, and what your specific regulator (RBI, DPDP, sector rules) actually demands.

Are on-prem open-weight models good enough versus frontier APIs?

For most business tasks, yes. Open-weight models like Llama, Mistral and Qwen have closed much of the gap and handle retrieval, extraction, classification and summarization well. Frontier APIs still lead on the hardest reasoning. The honest trade is a modest capability gap and real ops burden in exchange for full data control, worth it when the data genuinely cannot leave, over-engineering when it can.

Is the public API safe if the vendor promises not to train on my data?

Enterprise API tiers with contractual no-training and no-retention terms are a real, meaningful control, and for non-regulated or de-identified data they are often the right choice. What they do not change is jurisdiction, a foreign-controlled vendor still sits under its home country's laws. So it comes down to your data classification and whether sovereignty, not just privacy, is the requirement.