AI in business

Running sovereign AI yourself: models, hardware, effort

Florian Rocktäschel · Last reviewed: November 2026 · 12 minute read

In short

Between a US provider’s API and a server in your own rack there are at least three other options: a European model provider such as Mistral, a European cloud that serves open models for you, and a rented GPU server on which you run the model yourself. Self-hosting rarely pays off on cost. It pays off when you need to control the data, the model version and your exit. Then three questions decide the outcome: are you allowed to use the model (licence), does it fit in GPU memory (hardware), and who keeps it running at 3 a.m. (operations)? How much operations you need depends on the use: a local model for individual tasks runs with almost no attention, a service for many employees at once needs real operations.

What does "sovereign" mean for AI?

Nobody is fully independent. No company builds its own GPUs, and even an open model was trained by someone else. To me, sovereign means you know your dependencies, accept them deliberately and can unwind them if you need to.

That gives you four questions you can actually check:

  • Jurisdiction: which law applies to the provider, and who can demand access to your data?
  • Data flow: where are prompts and responses processed and stored, including logs and metadata?
  • Model version: do you decide when the model changes, or does the provider?
  • Exit: could you carry on elsewhere with the same or an equivalent model within three months?

What are the options, and how do they differ?

I distinguish five options. The providers in brackets illustrate the type; they are not recommendations.

1. A US provider’s API (such as Anthropic or OpenAI): the strongest models, nothing to operate, billed per token. The provider is also subject to US law (CLOUD Act), even when processing happens in Europe. 2. A European model provider’s API (such as Mistral): EU law, its own models, some with open weights. Mistral offers an EU endpoint that runs in data centres in EU and EFTA countries and costs 10% more than its global endpoint. 3. A European cloud serving open models (such as STACKIT AI Model Serving): you pay per token as with an API, but for open models hosted in a European data centre. You choose from the provider’s catalogue. 4. A rented GPU server (such as Hetzner or Scaleway): you run the model yourself while the data centre owns the hardware. Full control over model and version, and full responsibility for the software. 5. Your own server in your own or a colocation data centre: control down to the hardware, in exchange for purchase costs, power, spare parts and on-call duty.

Each step down gives you more control and more work. Model quality usually drops as well, because the strongest models are not released as open weights.

Which open models are you actually allowed to use in Europe?

“Open” does not always mean open source, so the licence comes before any hardware planning. Three examples from November 2026 show the range:

  • Apache 2.0, the simple case: Mistral Small 4, the Ministral 3 models, Google’s Gemma 4 and Alibaba’s open Qwen models. Commercial use, modification and redistribution are allowed.
  • Open weights with strings attached: Mistral Large 4, a model with around one trillion parameters, ships under the “Mistral Ecosystem Open-Weight License”. Commercial use is allowed, but selling the model to others as a service requires a separate licence from Mistral. Running it inside your own company is usually fine; offering it to your clients as a provider is not.
  • Excluded in Europe: Meta’s acceptable use policy for Llama 4 does not grant the licence rights for the multimodal models to individuals domiciled in the EU or companies with their principal place of business there. End users of a finished product are exempt. If you are an EU company planning to self-host, you are out.

Always check the licence at the source, on the developer’s model card. Repackaged versions, for example in hardware vendors’ containers, can add terms of their own.

How much GPU memory does a model need?

For a first estimate, a formula from Lenovo’s sizing guide is enough: parameters in billions, times bytes per parameter, times 1.2. Full precision (FP16) means 2 bytes per parameter, 8-bit means 1 byte, 4-bit half a byte. The factor of 1.2 covers loading overhead.

Model Licence Parameters (active) Memory at 8-bit Memory at 4-bit
Ministral 3 14B Apache 2.0 14 bn approx. 17 GB approx. 8 GB
Qwen3.6 27B Apache 2.0 27 bn approx. 32 GB approx. 16 GB
Gemma 4 31B Apache 2.0 31 bn approx. 37 GB approx. 19 GB
Mistral Small 4 Apache 2.0 119 bn (6.5 bn) approx. 143 GB approx. 71 GB
Mistral Large 4 Mistral licence with conditions approx. 1 trillion (approx. 50 bn) approx. 1,200 GB approx. 600 GB

These figures cover the model weights only. Two things come on top, and teams underestimate both.

The first is active parameters. Mistral Small 4 is a mixture-of-experts model: only 6.5 billion parameters do work for each token, which makes it fast. All 119 billion still have to sit in memory, because the router only picks the experts during the computation. Hugging Face explains this with Mixtral as the example: total parameters set the memory, active parameters set the compute. Size your hardware on active parameters and it will be too small.

The second is the KV cache, the working memory for requests in flight. It grows with the number of concurrent users and the context length. In Lenovo’s worked example, a 70-billion-parameter model serving 100 concurrent users with 8,000 tokens of context each needs about 262 GB for the cache alone, more than the 168 GB for the model itself.

A 4-bit version halves the memory but costs some quality, depending on the model. Only a test with your own questions tells you whether that is acceptable for your use case.

What does the hardware cost?

Market data, November 2026, excluding VAT:

Option Hardware Price Source
Hetzner GEX45 1 × RTX PRO 4000 Blackwell, 24 GB €214 per month + €209 setup hetzner.com
Hetzner GEX63 1 × RTX PRO 6000 Blackwell, 96 GB (Helsinki) €999 per month + €999 setup hetzner.com
Hetzner GEX131 1 × RTX PRO 6000 Blackwell, 96 GB (incl. Falkenstein, Germany) from €1,499 per month + €1,499 setup hetzner.com
Hetzner GEX430 4 × RTX PRO 6000 Blackwell, 384 GB in total €3,599 per month + €3,599 setup hetzner.com
Scaleway H100 1 × H100, 80 GB, per GPU hour about $3.50 (around $2,500 a month running continuously) computeprices.com price tracker
Buying an RTX PRO 6000 Blackwell one card, 96 GB roughly €11,000 to €15,000 at German retailers (mid-2026) PCGH price comparison

Read against the table above: a model in the 14 to 31 billion range runs on a single 96 GB card with room left for the cache. Mistral Small 4 just about fits on such a card at 4-bit; with many concurrent users you need two. Mistral Large 4 is a different league: its model card shows deployment examples on eight GPUs.

Owning the server adds power and cooling. Small and medium-sized industrial customers in Germany pay 17.9 cents per kWh on average in 2026, according to the German energy industry association BDEW. A server drawing one kilowatt around the clock uses 8,760 kWh a year, roughly €1,570 in electricity before the air conditioning starts.

Is self-hosting cheaper than an API?

Usually not, and that is the inconvenient number in this debate. An example: for the €1,499 a month of the GEX131, Mistral Small 4 via the EU endpoint gives you around six billion tokens a month, assuming four parts input to one part output. That is about 2,400 tokens per second, around the clock, every day. Only when your usage reaches that order of magnitude does the server beat the API for the same model. I walk through the full calculation, including caching, batch processing and staff time, in What an AI application costs to run.

So people who self-host do it for other reasons: the data must not leave the building, the model version has to stay fixed for years, or an outage at the provider is not an acceptable risk. Those are good reasons. Say them out loud instead of hiding them behind a cost calculation.

What does the CLOUD Act mean for AI services?

The US CLOUD Act of 2018 added a rule to the Stored Communications Act (18 U.S.C. § 2713): providers of electronic communication and remote computing services must disclose content and customer records in their possession, custody or control when ordered to, regardless of where the data is stored.

For AI services this means a US provider’s EU data centre changes the location, not the jurisdiction. “Processing in Europe” is data residency. It answers the question of where, not the question of who can demand access.

The other side belongs in the picture too. Transfers of personal data to the US are currently permitted under the EU-US Data Privacy Framework. The EU General Court dismissed a challenge in September 2025, and the appeal is pending before the Court of Justice as case C-703/25 P. And a European provider also needs a data processing agreement under Article 28 GDPR if you send it personal data.

European providers have small print as well. Mistral’s documentation states that its EU endpoint does not keep all control-plane data in the region: account details, API keys, billing and usage analytics may be handled elsewhere. For most companies that is acceptable. You just want to know before you write “everything stays in the EU” into a privacy notice.

This is not legal advice. Whether an option is permissible for your data is a question for your data protection officer or a lawyer.

What does it take to operate it yourself?

The hardware is the visible part. The larger part is the work an API provider otherwise does for you:

  • Setting up and maintaining an inference server such as vLLM or SGLang (both open source), plus drivers and the operating system.
  • Managing model updates: new versions arrive monthly, and each one has to pass your test set before it goes live.
  • Monitoring latency, utilisation, errors and memory. Without measurements you only learn from complaints that the server is at its limit.
  • Resilience: one server is a single point of failure. For a service your staff rely on, you need a second one or a fallback via an API.
  • Security: access control, logging, patching. An inference server is a server like any other and gets attacked like one.
  • On-call: who responds when the service is down on Monday morning?

This is where the most common misjudgement sits. If you write “sovereign” on the slide and then leave the server to a single person, you have only moved the dependency, from a vendor to an employee.

Which option fits which case?

Criterion US API European provider (API) European cloud, open models Rented GPU server Own server
Model quality highest high medium to high depends on hardware depends on hardware
Provider jurisdiction US (CLOUD Act) EU EU EU (data centre) your own
You control the model version no partly partly yes yes
Your operating effort low low low high highest
Cost at low usage low low low high (fixed costs) high (purchase)
Exit re-test your prompts easy with open models easy easy easy, hardware stays

One route I often find sensible: start with a European provider’s API and pick a model whose weights are open. That keeps the exit to self-hosting with the same model open without rebuilding the application. Mistral Small 4 is an example: the same model is available via API and as an Apache 2.0 download. How to build applications so that switching models is not a fresh start is the topic of Switching models without starting over.

And not every application needs the same route. Marketing copy can go through a US API; HR files and engineering data perhaps cannot. Sort your use cases by data before you choose a platform.

From practice

I run models myself, on local machines and on a laptop too. In many of my projects the data may not go to the cloud or to any outside provider at all, for example when I sort and analyse data sets for clients. So the model runs locally.

The effort is smaller than the list above suggests. For tasks like these, operation costs practically no person-days. The model runs in a Docker container and simply keeps running. You have to get the hardware right once. That can be a desktop machine, which ends up a few thousand euros more expensive than an office PC. For models up to about 27 billion parameters at 4 bit, around 16 GB of GPU memory according to the table above, a matching graphics card is enough. AI helps with the setup, by the way; for me that is standard.

The list above is for the other case: a service many employees rely on at the same time, one that has to scale. That is a topic of its own with effort of its own. I still think self-hosted models are the future, because there you decide for yourself and can see what happens inside the model.

Checklist before you decide

  • Which data flows into the application, and which of it must not leave the company?
  • Does the licence of the model you want stand in the way, for example through a clause for service providers or an EU exclusion?
  • How many people use it at the same time, and how long are contexts and answers? That sets your memory needs.
  • Do you have a test set of real questions to compare a smaller or quantised model against the large one?
  • Who runs the server, who covers for that person, and what happens during an outage?
  • At what volume does self-hosting pay off, and are you there?

How a prototype becomes a service that can carry this kind of operation is covered in From AI prototype to production. If you want to talk the decision through with someone who builds these platforms, that is what my page AI in your company is about.

Disclosure

As a contractor, I develop Kivvo, an AI platform operated in Germany by KI Partner Deutschland GmbH. That is why this article does not recommend any platform. The providers named are examples of a type.

Written for

IT leads, developers and executives who already use AI and now have to decide where their models should run.

In this article
Share this article

Frequently asked questions

Small models run on CPUs, but much more slowly. For a service several employees use at once, you will in practice need a GPU with enough memory for the model and the cache.

Transfers to the US are currently permitted under the EU-US Data Privacy Framework, and you need a data processing agreement either way. A European location does not change the fact that the provider is subject to the CLOUD Act. Whether that is acceptable for your data is a risk decision you should have reviewed legally.

As a rule of thumb: parameters in billions, times bytes per parameter, times 1.2. A 31-billion-parameter model at 8-bit needs about 37 GB, plus cache for concurrent requests. For mixture-of-experts models, count total parameters, not active ones.

No. Open weights can come with conditions, such as a clause for companies selling the model as a service or an exclusion for EU companies. What counts is the licence on the developer’s model card.

Rarely on cost alone, unless usage is very high. It is worth it when data must not leave the building or you need to fix the model version for a long time, and when someone can take on operations permanently. For individual tasks with confidential data, a local machine with a suitable graphics card is often enough, and it needs hardly any attention.

Sources

Last reviewed November 2026. Prices are market data as of the stated date, excluding VAT, without guarantee. Legal information without guarantee; not legal advice.

About the author

I am Florian Rocktäschel, an engineer from RWTH Aachen. I have worked with machine learning since 2013 and as a fractional CTO for mid-sized companies since 2018. Here I write down what I am asked again and again in this work.

More about me

Where does your technology stand?

The Technology Traffic Light asks 15 questions about AI, strategy, security, data, costs and team. Ten minutes, instant result, free of charge.

Start the Technology Traffic Light
Example of a Technology Traffic Light report (sample company)

Contact

open@bitflinx.de
+49 (241) 89 43 89 17

Locations

Aachen (Germany)
Bucharest (Romania)


© 2026 BITFLINX. All rights reserved.