A local LLM is a language model that runs on your own hardware, in your server room or on a workstation, without inputs ever leaving the building. Cloud AI is the opposite: you send your prompts to a provider whose model runs on the provider’s servers. Plenty of law firms, tax advisors, and financial services providers are choosing between these two paths right now, usually with the argument that “local is the safe option for data protection.”
That argument is half right. A local LLM solves one problem very elegantly, namely third-country transfers and the question of who at the provider can read along. At the same time it shifts a whole set of duties away from a corporation with a security department and onto you. This post shows both sides so you can make the decision with your eyes open.
What is a local LLM?
Technically, a local LLM is a downloaded open-weights model executed on your hardware through a runtime. Well-known open-weights model families include Llama from Meta, Mistral from France, Gemma from Google, and Qwen from Alibaba. Common runtimes are Ollama, llama.cpp, and vLLM.
One clarification matters here. “Open weights” means you may download the model and run it yourself. It does not automatically mean open source in the classic sense, because the licenses differ considerably and some carry usage restrictions. If you put a model into production, read its license once.
There is also a third option between local and cloud that often gets lost in the debate: an open-weights model hosted by a European provider. You do not run it yourself, but processing happens at a processor inside the EU. For many firms this middle path is the more practical one.
Does a local LLM solve the GDPR questions?
It solves some of them, and they happen to be the uncomfortable ones.
What genuinely goes away: if no data leaves the building, there is no third-country transfer under Chapter V of the GDPR. You need no standard contractual clauses and do not have to rely on the EU-US Data Privacy Framework. You need no data processing agreement for the model itself, because there is no processor. The question of whether the provider trains on your inputs does not arise. And for professionals bound by confidentiality the point is especially relevant: if nobody outside your firm gets access, there is no disclosure to a third party, which is what makes Section 203 of the German Criminal Code an issue in the first place.
What stays: pretty much everything else. You remain the controller under GDPR. You still need a legal basis under Article 6, you still have to enter the processing in your records of processing activities, and where the risk is high you still need a data protection impact assessment. Data subject rights, retention rules, and logging remain your job too. A local model is not a free pass under data protection law. It is a good answer to exactly one part of the question.
And one duty actually grows rather than shrinks: security of processing under Article 32 GDPR. With cloud AI, the provider carries a substantial share of the technical safeguards through data centre security, patch management, access controls, and certifications. When you run the model yourself, all of that lands on you.
The honest comparison
Privacy and confidentiality. Local wins, clearly. No transfer, no third party, no training question.
Security effort. Cloud wins. A locally operated model needs network segmentation, access control, logging, backups, and regular updates to the runtime. From my time in security practice I know the pattern: a server that gets set up once and then never touched again is the weakest link on the network twelve months later. Running a local LLM means running a piece of infrastructure, not just installing software.
Cost. Mixed, and distributed differently than most people expect. Cloud AI costs a predictable amount per user per month. Local costs hardware up front (a usable GPU server runs in the four to low five figure range), then electricity, then above all staff time. The biggest line item is rarely the metal, it is the person who maintains the system. If you do not have that person, you buy them in, and the cost advantage usually disappears.
Quality. This is where the picture has shifted a lot in the past two years. Open models are good enough for summaries, drafts, translations, and standard correspondence in many cases. For complex reasoning across long documents, the large commercial models still lead. The honest advice: test on your own real tasks, not on benchmark tables.
Integration. Cloud wins, especially if you already work in Microsoft 365. Copilot sits inside Word, Outlook, and Teams. A local model starts life in a chat window, and everything beyond that is a project.
Availability. Cloud wins. If your own server goes down, the AI is gone until somebody fixes it. For a tool used in daily work, that is worth discussing before you commit.
Who should run a model in-house?
In my experience the decision rarely turns on data protection. It turns on who takes over operations.
A local LLM fits if you have IT competence in house or an IT provider who already looks after your servers, if you work with especially sensitive material (professional secrets, Article 9 data, mandates with their own confidentiality agreements), if the use cases are clearly bounded, and if you are prepared to fund operations permanently. Not as a project, but as a standing line item.
Cloud AI fits if you have no IT of your own, if you want broad everyday use rather than a few special cases, if you work in Microsoft 365, and if you need results quickly. The road there runs through a business contract with a data processing agreement, a deliberate region choice, and clean settings, of the kind I described in the Copilot settings that matter for GDPR.
The middle path fits if you want EU processing but no server operations of your own. An open-weights model hosted by a European provider, with a processing agreement and an EU region, delivers a large share of the privacy benefit without the operational load. A processor is still involved, so the usual checks on data residency apply.
Many firms end up running both: cloud for broad daily work, local for the one use case where genuinely nothing may go outside. That is not indecision, it is sensible risk distribution.
The most common mistake in thinking
The mistake I meet most often goes like this: “We will use a local model, so we do not need rules.” The opposite is true. Because no provider sets limits for you anymore when you run it yourself, your own rules have to do the work: who may use the system, which data may go in, how long chat histories are kept, who sees the logs, how outputs get checked before use. All of that belongs in the same AI acceptable use policy you need for cloud tools.
On top of that, a local model still makes mistakes and still invents content. The duty to have a human check the output does not disappear with the server location. And the AI literacy obligation under the EU AI Act applies regardless of where the model does its computing.
The next step
Local, cloud, or the middle path, the groundwork is identical in all three cases: clarify the use cases, establish the legal basis, set up the documentation, give the team its rules. If you want to know where you stand on that, take my free AI compliance check. It takes 2 minutes and shows you which questions are still open before you commit to a path.
Jose Lugo is a CISSP-certified AI compliance consultant. He advises law firms, tax advisors, and financial services providers in Germany on GDPR-compliant AI adoption.