What is locally hosted AI?
Locally hosted AI runs selected models on infrastructure you control, rather than sending every prompt to a public AI service. The intended design keeps inference and the agreed document-processing pipeline within your local environment.
That can give your business more control over prompts, source documents, access and retention. It also creates responsibilities for hardware, permissions, model selection and maintenance. We scope the capability around a defined business use case and test its behaviour before relying on it.
A private assistant for internal information
A local assistant can help staff find relevant information in approved policies, manuals, procedures and other internal documents. A retrieval workflow searches the selected material and provides relevant extracts to the model when it answers a question.
The aim is a useful answer connected to its source material. Document search does not require the model to be trained on every business file, and the presence of a citation does not prove the answer is correct. Users should check the source for important decisions.
Examples to assess in a pilot include:
- Finding an approved procedure and the document it came from.
- Summarising selected internal material for a member of staff who already has access.
- Drafting a response using authorised reference documents for a person to review.
- Helping a team navigate its own operating knowledge while an external AI service is unavailable.
Each is a proposed use case to validate against your data, rather than a claim that every document or workflow will work without configuration.
Keep permissions and data movement visible
Copying documents into an AI index creates another place where their contents may be exposed. The design must account for the source files, extracted text, embeddings, conversation history, logs and backups.
We agree which users may access which material, how those rules reach the retrieval layer and what is retained. A broad shared index must not silently give a user access to documents they could not otherwise read. Changes and deletions in the source also need an agreed treatment in the AI workspace.
Local hosting alone is not proof that nothing leaves the premises. Model downloads, update checks, telemetry, external tools and remote support need reviewing. An offline-capable design is tested with its external connection removed, using the complete workflow rather than only the model server.
Open-source software, with model licences checked separately
Established inference software such as llama.cpp and LocalAI provides options for locally operated services. Both projects publish their software under the MIT licence; the particular components are selected for the deployment.
Model weights have their own licences and usage conditions. An openly downloadable model is not automatically free of restrictions or equivalent to FOSS software. We consider the model’s permitted business use, hardware requirements and suitability for the task separately from the software running it.
You should have a documented route to export your source data and agreed configuration. Model and software choices should support a practical handover rather than unnecessary dependence on one appliance supplier.
Hardware sized for the workload
The right specification depends on model size, context length, document volume, concurrent users and acceptable response time. Some workloads can run on a CPU; others benefit from a suitable GPU and substantially more memory.
Start with a measured pilot using representative questions and authorised sample documents. Agree quality checks and capacity targets before specifying the production hardware. Local AI may be scoped alongside the continuity appliance or as a separate deployment; capacity and isolation requirements decide the design.
Local AI, cloud outages and data sovereignty
An appropriately prepared local AI service can reduce dependence on an external inference API for its agreed tasks. It still needs working power, hardware, the local network, authentication and the source data it uses.
It does not restore Outlook, replace Teams or automatically recreate a cloud application. Its role is to make selected local information more usable within a wider continuation plan.
For the wider decisions about location, access and processing, read our data sovereignty guide. The ICO’s AI security and data-minimisation guidance is a useful reference when the proposed workflow involves personal information.