Running Models Locally

Running Models Locally

You don't need a cloud API to use a language model. With the right tools, you can run capable models on your own hardware — a laptop, a home server, or a workstation with a consumer GPU. This opens up use cases where privacy, cost, or latency matter.

Ollama: The Easiest Path

Ollama has made local model deployment trivially easy. Install it, then a single command downloads and runs a model.

Ollama handles model downloading, format conversion, and serving — exposing a local API compatible with the OpenAI SDK. You can point most tools that support the OpenAI API at a local Ollama instance with minimal changes.

Ollama uses llama.cpp under the hood, which enables efficient CPU inference — you don't even need a GPU, though one helps dramatically with speed.

As of October 2026: popular models to try locally include Qwen3, DeepSeek-R1 distills, Llama 4 Scout, and Gemma 3 — ollama run qwen3 takes seconds to set up.

Hardware Requirements

Model size determines what hardware you need:

CPU-only (any modern computer): small parameter-count models run at modest speeds on a modern CPU. Usable but slow for interactive use.

Consumer GPU (mid-range VRAM): comfortably runs small-to-medium models at reduced precision, tens of tokens per second.

High-end GPU (high VRAM): runs larger dense models, or reduced-precision versions of bigger ones still. Near-API-quality output, fast.

Apple Silicon: unified memory architecture means RAM is shared between CPU and GPU, so a Mac with enough unified memory can run very large models efficiently. Excellent local inference platform.

Quantisation: models are often distributed in quantised form (4-bit, 8-bit) which reduces memory requirements by 2–4x with small quality loss.

Mixture of experts (MoE): many current local-friendly models use a mixture-of-experts architecture — a large total parameter count split into smaller "expert" sub-networks, with only a handful active for any given token. This changes the memory maths above: you generally need enough memory to hold the total parameter count, even though only the smaller active parameter count is actually computed per token — so an MoE model can respond faster than a dense model of the same total size, without necessarily needing less memory to load.

As of October 2026: current consumer GPUs for local inference include the RTX 50-series (e.g. RTX 5070/5080/5090) with 12–32GB VRAM, with high-VRAM workstation cards like the RTX Pro 6000 available for larger models. On Apple Silicon, the M4 and M5 generations (Pro/Max/Ultra) offer unified memory up to several hundred GB on the top configurations — enough to run very large models, including large MoE models, locally.

When to Run Locally

Privacy-sensitive applications: medical records, legal documents, personal data — nothing leaves the device.

High-volume, lower-stakes tasks: if you're processing thousands of documents and frontier-level quality isn't required, local models eliminate API costs.

Offline use: no internet connection required after the initial download.

Experimentation: try new models, fine-tunes, and configurations without worrying about per-token costs.

Latency-sensitive edge cases: a local model on a fast machine can beat API round-trip time for some use cases.

The trade-off: local models are still typically behind the very best frontier API models on the most complex reasoning tasks. For high-stakes, complex reasoning, frontier API models still tend to win. For coding assistance, summarisation, classification, and many other tasks, local models are competitive.

As of October 2026: the best open-weight models runnable locally are closely competitive with frontier closed models on many benchmarks, though the largest of them need workstation or multi-GPU hardware to run at full precision.