Run LLMs Locally

Why run locally (and why not)

Three reasons pull people to local models. Privacy: the text never leaves your machine, which matters for personal notes, client data, medical or legal material, or anything you simply don't want sitting on someone's server. Cost: once the model is downloaded, inference is free — no per-token bill, no metering, however much you generate. Control: it runs offline, it won't be deprecated out from under you, and nobody rate-limits or changes the model behind your back.

Now the honest half. A model you can run on a laptop will not match a frontier hosted model on the hardest reasoning and coding tasks — the gap is real, even if it keeps shrinking. You trade some quality and speed for privacy and cost. You'll also do a little setup, and top-tier hardware isn't cheap. For summarizing, drafting, classification, coding help, and private chat, a good local model is more than enough. For your most demanding work, you may still reach for a hosted model. Knowing which job is which is most of the skill.

The hardware reality check

One number governs almost everything: memory. On an NVIDIA or AMD GPU that's VRAM; on an Apple Silicon Mac it's unified memory shared with the system. The model's weights have to fit, or performance falls off a cliff as it spills to slower storage. Here's the practical map for 4-bit models: 8 GB comfortably runs 7–8B-parameter models; 24 GB is the floor for 30B-class models; 40 GB and up is where 70B territory lives unless you quantize hard.

No dedicated GPU? You can still run smaller models on CPU and system RAM — llama.cpp was built for exactly that, and a modern laptop with 16 GB of RAM runs small models fine, just slower. Apple Silicon is a quiet winner here: because memory is unified, a Mac with 32–64 GB can hold models a similarly-priced PC can't fit in its GPU.

The tools that run models

Five runtimes cover essentially every situation, and they overlap less than you'd think. Ollama is the one-command default for solo developers — install it, type one line, and you're chatting. LM Studio wraps a polished desktop GUI around the same idea, plus a headless server mode; it's the friendliest on-ramp if you'd rather click than type. llama.cpp is the C/C++ engine underneath much of this ecosystem (Ollama and LM Studio build on its ggml library); reach for it directly when you need to run on unusual or CPU-only hardware. vLLM is the production server — it uses PagedAttention to serve many concurrent users fast, and it's what you'd deploy to a team or an app. mlx-lm is the Apple Silicon specialist, tuned to wring throughput and fine-tuning out of a Mac.

Choosing is easy once you name the goal: running a model in the next five minutes → Ollama or LM Studio; running on hardware nobody else supports → llama.cpp; serving a hundred people at once → vLLM; squeezing the most out of a Mac → mlx-lm. Start with Ollama and graduate as your needs grow.

Choosing an open-weight model

"Open-weight" means the trained weights are published so you can download and run them yourself — distinct from "open-source," which would also include training code and data. The field moves monthly, so learn the families rather than memorizing version numbers. As of mid-2026 the ones worth knowing are Google's Gemma, Alibaba's Qwen, DeepSeek, Meta's Llama, Moonshot's Kimi, and Zhipu's GLM, alongside Mistral's models.

They split into two tracks. The small, laptop-friendly models — Gemma's smaller variants, small Qwen models, and similar — are what you'll actually run at home, roughly 4B to 30B parameters. The giant mixture-of-experts models like the largest GLM, Kimi, and DeepSeek releases are datacenter-scale and not meant for a single machine. Pick a small model in a family with an active release cadence, and always check its license before any commercial use — terms vary by family and by version, from permissive Apache-style licenses to custom terms with conditions.

Browse current models at ollama.com/library and Hugging Face, and let our AI Model Picker narrow the field for your use case.

Quantization basics

A model's weights are trained in 16-bit numbers, but you rarely need that much precision to run it. Quantization stores each weight in fewer bits — commonly 4 — which shrinks the file and the memory footprint by roughly a factor of four, with only a small quality cost. It's the single technique that lets a big model fit on a small machine. The common local format is GGUF, and within it you'll see labels like Q4_K_M, Q5_K_M, and Q8_0.

Decode the label: the number is the bit-width, K means the smarter block-wise "k-quant" method, and the suffix is the size/quality tier (S small, M medium, L large). Q4_K_M — about 4.5 bits per weight — is the widely-recommended default: nearly full quality at a fraction of the size. Go higher (Q5, Q6, Q8) if you have memory to spare and want the last bit of fidelity; go lower only when you're desperate to fit.

Your first project

Time to run one. Install Ollama, then pull and run a small model. Ollama downloads a sensibly-quantized GGUF automatically, so you don't have to think about quant labels on your first try.

# 1. Install Ollama from ollama.com, then in a terminal:
ollama run gemma3
#    (browse ollama.com/library for the current model names —
#     the catalog changes often; any small model works to start)

# 2. It downloads once, then you're chatting right in the terminal.

Once it's running, the real unlock is the local API. Ollama serves an HTTP endpoint on localhost:11434, so your own scripts can call the model exactly like a cloud API — but private and free:

# Call your local model from anything that speaks HTTP:
curl http://localhost:11434/api/generate -d '{
  "model": "gemma3",
  "prompt": "Explain quantization in one sentence.",
  "stream": false
}'

That endpoint is your foundation for everything else — summarizers, chatbots, and agents that never send a byte to anyone else's server.