Self-hosted models
Run open LLMs like Llama or Mistral on your own server with tools like Ollama or TGI.
Last updated
Analogy
Running your own Postgres instead of a managed one: full control, full responsibility.
In an agent
Data never leaves your infrastructure, which matters for client code, health or finance data.
Practice
Interview questions
- When would you use an open-source model?
- High-volume narrow tasks like embeddings and classification.
- Strict privacy requirements, or when cost per call matters.
- In DevPilot, Hugging Face models do embeddings and PR classification locally in Node.
Questions people ask
How do I self-host an open-source LLM?
Use a serving tool such as Ollama for simple setups or Text Generation Inference for production, and run an open model like Llama or Mistral on your own machine or server. Your app then calls it over HTTP, much like a hosted API. You take on the hardware, updates and scaling yourself.
Why would a company self-host its models?
Mainly data control: prompts and documents never leave the company's own infrastructure, which matters for client code, health or financial data. At very high volumes it can also be cheaper. The cost is responsibility for GPUs, uptime, security updates and quality, much like running your own database.