An on-premise LLM is a large language model that runs on your own servers, or in a private cloud you control, instead of being called through a public AI service. Prompts, documents and answers never leave an environment you control. A workable deployment has four parts: an open-weight model sized for the job, enough GPU memory to hold it, a serving layer that handles many users at once and a RAG pipeline that reaches documents according to each user's permissions.
Key takeaways
- An on-premise LLM is a strong alternative to cloud APIs where data must stay in-house; it is not the right answer for every task.
- Model size and numerical precision decide how much GPU memory you need; 4-bit quantisation cuts weight memory to roughly a quarter of FP16.
- Ollama or llama.cpp is fine for a single-user trial; dozens of concurrent users need a serving layer such as vLLM.
- Installing the model is only half the work: without RAG, access control and logging the system is neither useful nor safe.
- According to Verizon's 2026 DBIR, 67% of AI users reach these services through non-corporate accounts; a local model gives that demand a governed channel.
Why do public AI services put company data at risk?
Your staff are already using AI; the question is where, and under which account. Verizon's 2026 Data Breach Investigations Report found that 45% of employees are now regular AI users on corporate devices, up from 15% a year earlier, and 67% of users access AI services with non-corporate accounts. Proposals, contracts and source code end up in accounts the company never sees, as we explain in ChatGPT and company data security; see also shadow AI.
Privacy concerns also slow AI adoption. Eurostat reports that among EU enterprises that considered AI in 2025 but did not adopt it, 48.83% cited data protection and privacy concerns and 70.89% a lack of relevant expertise. A local model answers the first concern; the second needs a sound deployment and someone to run it.
There is a legal angle too. Sending personal data to a service hosted abroad can count as a cross-border transfer under Turkish data protection law (KVKK); such a transfer must rely on one of the conditions in Article 9 of the law, such as an adequacy decision, appropriate safeguards like standard contractual clauses, or limited exceptions. We explain that framework in KVKK cross-border data transfers and AI and KVKK.
When is an on-premise LLM the right choice, and when is it not?
Running models locally pays off when data is sensitive, usage is steady and connectivity cannot be taken for granted. For occasional tasks needing the largest, newest models, a cloud API is usually cheaper. Many organisations end up with a hybrid of both.
| Criterion | Cloud API | On-premise LLM (your servers) | Private cloud / hybrid |
|---|---|---|---|
| Where the data sits | Provider's infrastructure | Your own network | An isolated environment you rent |
| Up-front investment | Low | GPU server plus set-up effort | Medium |
| Cost as usage grows | Rises with every request | Close to flat up to hardware capacity | Depends on capacity |
| Model quality | Latest commercial models | Open-weight models | Can combine both |
| During an internet outage | Stops working | Keeps working | Depends on design |
| Expertise required | Low | GPUs, serving software, security | Medium to high |
| Cross-border transfer | Contract and legal basis may be needed | Data stays in the country and in-house | Depends on region chosen |
A practical rule: personnel files, health data, customer contracts, formulations and pricing go to the local model; general drafting, translation and public research can go to the cloud. To put that split in writing, start with our guide to a company AI acceptable use policy.
What hardware does an on-premise LLM need?
The deciding resource is not the CPU but GPU memory (VRAM). The space taken by the model weights is roughly the parameter count multiplied by bytes per parameter: 2 bytes at FP16, 1 byte with 8-bit quantisation and about half a byte with 4-bit. On top of that, allow at least 20–30% for the context window (the KV cache) and runtime overhead; with long contexts and many concurrent users the margin grows much larger.
| Model size | FP16 weight memory | 4-bit weight memory | Typical hardware (quantised) | Suitable work |
|---|---|---|---|---|
| 7–8 billion parameters | ~16 GB | ~5 GB | A single workstation GPU | Summaries, classification, simple document Q&A |
| 13–14 billion | ~28 GB | ~9 GB | One 24 GB card | Enterprise RAG assistant, email drafts |
| 30–34 billion | ~65 GB | ~20 GB | One 48 GB card or two 24 GB cards | Longer documents, multi-step answers |
| 70 billion | ~140 GB | ~40 GB | Two 48 GB cards or data-centre GPUs | Complex reasoning, many concurrent users |
These figures are approximate and cover weights only; the hardware column assumes 4- or 8-bit quantisation, and FP16 needs roughly two to four times the memory. Long documents and simultaneous users enlarge the KV cache, so plan capacity around your busiest hour. The server also needs fast NVMe storage, ample RAM and a UPS.
Quantisation is not a free discount. A 4-bit model is good enough for most tasks, but quality loss shows sooner in arithmetic, legal interpretation and morphologically rich languages such as Turkish. Make the call by measuring against the evaluation set described below.
Which model and which software should you choose?
On the model side, open-weight families such as Llama, Mistral, Qwen and Gemma are the most common choices. "Open-weight" does not always mean "open source": some licences restrict commercial use, user numbers or fields of use. Read the licence with your legal team and test candidates with your own questions.
The software has two layers. For trials and single users, Ollama or llama.cpp installs in minutes and even runs on a laptop. A system serving dozens of concurrent users needs a serving layer that batches requests, such as vLLM or SGLang; both expose an OpenAI-compatible API, so existing applications connect with few changes.
The model does not know your documents out of the box. To answer from them with citations you need a RAG architecture and a vector database that finds passages by meaning.
If the model has to learn your organisation's vocabulary or a specific task, fine-tuning comes in. We compare the options in our custom AI model training guide.
How do you deploy an on-premise LLM step by step?
This sequence avoids the usual mistakes made between trial and production.
- Define the use case and data class. Which team will ask which questions of which documents? The sensitivity of the data decides between local and cloud.
- Build an evaluation set. Write 50–100 real questions with correct answers; every model and setting is measured against it.
- Choose the model and check the licence. Compare two or three candidates on the evaluation set and confirm the terms for commercial use.
- Size the hardware. Work out GPU memory from model size, quantisation level, context length and peak concurrent users.
- Stand up the serving layer. Publish the model through a server such as vLLM, expose the API only on the internal network and add authentication.
- Connect documents through RAG and enforce permissions. Users should only get answers from documents they are allowed to see; the principles are in permission-aware AI assistants.
- Add security and logging. Keep prompts and answers for a period that matches your data policy, and defend against instructions hidden in documents with prompt injection controls.
- Pilot with a small team. Measure accuracy, speed and how often the assistant says it cannot find an answer; add poor answers to the evaluation set.
- Plan for operations. Assign owners and a schedule for model updates, GPU monitoring, backups and indexing new documents.
What are the most common mistakes in on-premise deployments?
The first is buying hardware for a single-user demo. A model that is instant on a laptop can leave forty simultaneous users waiting for minutes. Capacity should be calculated from the number of concurrent requests in the busiest hour.
The second is skipping access control because "the data stayed inside". IBM's 2025 Cost of a Data Breach report found that 13% of organisations reported breaches of AI models or applications, and 97% of those lacked proper AI access controls. A local model can just as easily summarise the payroll spreadsheet for someone who should never see it.
The third is trusting the model blindly. Open-weight models can also produce information that is not in the source, so citing sources and saying "this is not in the documents" must be designed in. We gather practical safeguards in reducing AI hallucinations.
How can Digital Bridge help with an on-premise LLM?
Through its enterprise LLM assistant service, Digital Bridge can deploy an assistant built on open-source (open-weight) language models entirely on your own servers. It answers from your procedures, manuals and contracts using RAG, shows which document section each answer relies on, and only draws on documents the user is authorised to see. Reports on frequent and unanswered questions expose missing or outdated procedures.
When the model needs to learn your terminology, document formats or a specific classification task, custom AI model training takes over, and the trained model is delivered to run on your own servers. Connecting the local model to existing systems such as ERP, CRM or your custom software falls under our AI integration service. Staff then ask from screens they already use.
You can start before buying hardware: a short trial against an evaluation set shows which model size is enough. For working out the business case, see measuring AI project ROI.
What is the next step towards an on-premise LLM?
Pick one job first: the most repeated question, the most searched-for document or a process whose data must not leave the building. Collect sample questions and documents for it; with those, model size, hardware and timeline become concrete.
To discuss an AI assistant that runs on your own servers, write to us via our contact page or call 0850 346 41 09 (+90 850 346 41 09 from abroad). For more guides, browse our AI articles, and check the LLM entry in our glossary for the basics.