Running an LLM on your own machine comes down to two hardware numbers and one file decision. The numbers are memory capacity, which decides whether a model runs at all, and memory bandwidth, which decides how fast it writes. The file decision is which GGUF quantization to download, and for most people the answer is Q4_K_M: about 4.8 bits per weight, so an 8-billion-parameter model becomes a 4.9 GB file that fits an 8 GB card with room for context. Generation speed has a ceiling you can compute before spending a dollar: bandwidth divided by model size. An RTX 4090 moves 1,008 GB/s, so a 4.9 GB model tops out near 205 tokens per second and lands around 140 in practice. A desktop CPU on dual-channel DDR5-5600 moves 89.6 GB/s and lands near 10. Everything below is detail on those two numbers.
Key Takeaways
- Capacity decides yes or no; bandwidth decides how fast. Nothing else about the hardware matters as much, including advertised teraflops.
- Q4_K_M is the default for a reason. A larger model at Q4 usually beats a smaller one at Q8 of the same file size, until about Q3.
- The KV cache is what surprises people. An 8B model at full 128k context needs 16 GiB of cache, over three times the weights.
- Compute your ceiling before buying. Bandwidth divided by file size, times 0.7, is close enough to plan with.
- Cost is rarely the reason to go local. Data control, offline operation, and freedom from rate limits are.
This assumes a terminal and some exposure to an LLM API. For the friendly on-ramp, the Ollama guide covers the ten-minute version and the open-source model guide covers which weights are worth downloading. This page is the layer underneath both: why a model is slow, why it will not load, and what to do about it.
The Two Hardware Numbers That Decide Everything
Generating a token requires reading every active weight out of memory exactly once. That single fact makes generation a memory-bandwidth problem, not a compute problem. The GPU's arithmetic units sit mostly idle while the memory bus works, which is why an expensive card with slow memory loses to a cheap one with fast memory.
Two phases run with completely different characters. Prefill processes your prompt: many tokens go through the network together, the math becomes matrix-by-matrix, and the compute units get used properly. Prefill on a modern GPU runs in the thousands of tokens per second. Decode produces the answer one token at a time, the math collapses to matrix-by-vector, and each token drags the whole model across the memory bus again. Decode is the number you feel while watching text appear.
Capacity tells you what will run; bandwidth tells you how fast. Here is the hardware you are likely choosing between.
| Hardware | Usable memory | Bandwidth | Practical ceiling |
|---|---|---|---|
| Desktop CPU, dual-channel DDR5-5600 | System RAM (cheap, large) | 89.6 GB/s | Small models only |
| RTX 4060 Ti 16 GB | 16 GB | 288 GB/s | 8B comfortably, 14B tight |
| Apple M4 Pro | Up to 64 GB unified | 273 GB/s | Runs big, generates slowly |
| RTX 3090 / 4090 | 24 GB | 936 / 1,008 GB/s | Up to 32B at Q4 |
| Apple M4 Max | Up to 128 GB unified | 546 GB/s | 70B at Q4, single stream |
| Apple M3 Ultra | 96 GB and up, unified | 819 GB/s | 70B comfortably |
| RTX 5090 | 32 GB | 1,792 GB/s | Fastest consumer decode |
The One-Line Rule
If a model fits entirely in the fastest memory you own, it will be fast. If any part of it spills to slower memory, the slow part sets the pace for the whole thing. Capacity is a cliff, not a slope.
Pick a Runner: llama.cpp or Ollama
llama.cpp is the C++ inference engine nearly everything else is built on. Ollama packages the same GGML core behind a model registry, an HTTP server, and automatic memory management. Start with Ollama if you want a chat model running in ten minutes; use llama.cpp directly when you need to control offload, context, cache type, or benchmark anything.
Ollama first, because it is three commands.
# macOS and Linux
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b --verbose
# --verbose prints the numbers that matter after every reply:
# prompt eval rate: 1240.55 tokens/s <- prefill
# eval rate: 94.31 tokens/s <- decode, the one you feel
ollama ps # shows whether the model landed on GPU or partly on CPU
That ollama ps line is the one people skip. Its PROCESSOR column reads 100% GPU, 100% CPU, or a split like 62%/38% CPU/GPU. A split is almost always why a model feels slow, and it is fixed by shrinking the model or the context, never by waiting.
llama.cpp takes a build step and gives back every knob. The old Makefile is deprecated and now errors on purpose, so use CMake.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# NVIDIA. On a Mac, drop the flag; Metal is on by default.
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
# Pull straight from Hugging Face with repo:quant shorthand
./build/bin/llama-server \
-hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M \
-ngl 99 -c 8192 --host 127.0.0.1 --port 8080
# -ngl 99 put every layer on the GPU (99 just means "all of them")
# -c 8192 context window; this is the KV cache size lever
# Open http://localhost:8080 for the built-in chat UI, or POST to
# /v1/chat/completions with an OpenAI-shaped request body.
Both servers speak the OpenAI chat-completions shape, so any client library works by pointing at a different base URL. Ollama listens on 11434, llama-server on whatever --port you pass.
| Runner | What it is for | Choose it when | Cost of choosing it |
|---|---|---|---|
| Ollama | One user, minimum friction | A local chat model or coding backend, today | Defaults are hidden; tuning means env vars and Modelfiles |
| llama.cpp | The engine, every flag exposed | CPU offload, exact context control, real benchmarks | You manage builds, files, and flags yourself |
| LM Studio | Desktop app over the same GGUF files | You prefer a GUI for browsing quants | Closed-source; harder to script |
| vLLM | Many concurrent users, datacenter GPUs | Dozens of simultaneous requests, model fits VRAM | No CPU offload, GPU only, heavier to run |
| MLX | Apple's array framework | Mac-only work where prefill speed matters | Mac only, fewer models than GGUF |
GGUF Quantization Levels and What Each One Costs
Quantization stores each weight in fewer bits. GGUF's k-quants split tensors into 256-value super-blocks with per-sub-block scales, which is why effective bits per weight are fractional rather than round. The naming reads Q[bits]_K_[mix], where the trailing S, M, or L says how many bits the important tensors keep.
File size is the honest source of truth: divide bytes by parameter count and multiply by eight to get the real bits per weight for the file you downloaded. The table uses an 8B model to make sizes concrete.
| Quant | Bits per weight | 8B file size | Quality | Use it when |
|---|---|---|---|---|
| Q8_0 | ~8.5 | 8.5 GB | Indistinguishable from fp16 | Comparing quants; almost never for daily use |
| Q6_K | ~6.6 | 6.6 GB | No loss anyone can point to | Memory is free and speed does not matter |
| Q5_K_M | ~5.7 | 5.7 GB | Very close to the original | Code and structured output, when it fits |
| Q4_K_M | ~4.8 | 4.9 GB | Small, rarely noticed loss | The default. Start here. |
| IQ4_XS | ~4.3 | 4.4 GB | Near Q4_K_M, needs imatrix | You are 500 MB short of fitting |
| Q3_K_M | ~3.9 | 4.0 GB | Visible drift on hard prompts | Squeezing a bigger model into the card |
| Q2_K | ~3.0 | 3.1 GB | Repetition, dropped instructions | Experiments only |
Two families exist. K-quants (Q4_K_M and friends) are the workhorses. I-quants (IQ4_XS, IQ3_M) squeeze harder using an importance matrix, computed by running calibration text through the model to learn which weights deserve their bits. I-quants hold quality better at low bit counts but cost more compute per token, so on a slow CPU they can finish behind a larger k-quant.
Bigger Model, Lower Quant: The Rule and Its Limit
Given a fixed budget of gigabytes, a larger model at a lower quantization normally beats a smaller model at a higher one. A 14B at Q4_K_M (9.0 GB) will generally outperform an 8B at Q8_0 (8.5 GB) on reasoning tasks. The rule breaks below roughly Q3, where quantization damage stops being noise and starts being behavior. Do not run a 70B at Q2 to feel like you have a 70B.
You can measure the damage rather than trusting a table. Perplexity on a fixed text is the standard proxy, and llama.cpp ships the tool.
# 1. Convert Hugging Face weights to a 16-bit GGUF
python convert_hf_to_gguf.py ./Llama-3.1-8B-Instruct \
--outfile llama31-8b-f16.gguf --outtype f16
# 2. Quantize it
./build/bin/llama-quantize llama31-8b-f16.gguf llama31-8b-Q4_K_M.gguf Q4_K_M
# 3. Score both on the same text and compare
./build/bin/llama-perplexity -m llama31-8b-f16.gguf -f wiki.test.raw -ngl 99
./build/bin/llama-perplexity -m llama31-8b-Q4_K_M.gguf -f wiki.test.raw -ngl 99
# Lower is better. Read the gap as a percentage, not an absolute:
# a fraction of a percent is invisible, several percent is not.
Perplexity is blunt. It scores next-token prediction on generic text and can miss damage to instruction-following or JSON validity, which is exactly what breaks first in an application. With real work at stake, run a task-shaped comparison instead, the way the LLM evaluation guide lays out.
How Much VRAM Do I Need? The Actual Math
Three things occupy memory: the weights, the KV cache, and about a gigabyte of runtime overhead for compute buffers and the CUDA or Metal context. Weights are easy to predict. The KV cache catches people, because it grows linearly with context length and the defaults are enormous.
Weights first: bytes = parameters × bits_per_weight / 8. An 8.03B model at 4.85 bits per weight is 4.87 GB, matching the file you download.
The KV cache stores one key vector and one value vector per token, per layer, for every key-value head. Grouped-query attention, which every current model uses, cuts this sharply by sharing key-value heads across query heads.
def kv_cache_bytes(n_layers, n_kv_heads, head_dim, seq_len, bytes_per_elem=2):
"""2 for K and V. bytes_per_elem: 2 = fp16, 1 = q8_0."""
return 2 * n_layers * n_kv_heads * head_dim * seq_len * bytes_per_elem
# Llama 3.1 8B: 32 layers, 8 KV heads, head_dim 128
per_token = kv_cache_bytes(32, 8, 128, 1) # 131,072 B = 128 KiB
kv_cache_bytes(32, 8, 128, 8192) / 2**30 # 1.0 GiB at 8k context
kv_cache_bytes(32, 8, 128, 131072) / 2**30 # 16.0 GiB at full 128k
# Llama 3.3 70B: 80 layers, 8 KV heads, head_dim 128
kv_cache_bytes(80, 8, 128, 8192) / 2**30 # 2.5 GiB at 8k context
Look at the third line again. An 8B model advertising 128k context needs 16 GiB of cache to actually use it, more than three times the weights. That is why a model which loads fine at defaults suddenly spills to CPU when someone raises the context, and why raising context is the most expensive change you can make.
Set context to the model's maximum
Llama 3.1 8B at Q4_K_M with 128k context: 4.9 GB of weights plus 16 GiB of cache plus overhead is roughly 22 GB. It will not fit a 12 GB card, and half of it lands on the CPU.
Set context to what you send
The same model at 8,192 tokens: 4.9 GB plus 1 GiB plus overhead is roughly 7 GB. It fits an 8 GB card entirely on the GPU and runs at full speed.
When long context is genuinely needed, quantize the cache instead of shrinking it. Both runners support it, and the quality cost is far smaller than spilling to system memory.
# llama.cpp: q8_0 halves cache memory, q4_0 quarters it.
# Flash attention is required for quantized cache.
./build/bin/llama-server -m model.gguf -ngl 99 -c 32768 \
-fa --cache-type-k q8_0 --cache-type-v q8_0
# Ollama: same idea, via environment
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
One caution: quantized cache silently falls back to fp16 on architectures that do not support it, so confirm memory actually dropped rather than assuming the variable took effect. Newer llama.cpp builds also want -fa on rather than a bare -fa; check --help for the build you compiled.
How Many Tokens per Second to Expect
Divide memory bandwidth by model file size for the theoretical ceiling, then take 60 to 80 percent of it. Real systems lose the rest to KV cache reads, attention overhead, kernel launch latency, and imperfect memory access patterns. The estimate is close enough to decide what to buy.
Every figure below is derived from published bandwidth specifications at a 70 percent efficiency factor, not measured on a bench. Treat them as planning numbers, then measure your own machine.
| Setup | Model | Ceiling | Expect around |
|---|---|---|---|
| CPU, DDR5-5600 dual channel | 8B Q4_K_M | 18 t/s | 10–12 t/s |
| RTX 4060 Ti 16 GB | 8B Q4_K_M | 59 t/s | 40–45 t/s |
| Apple M4 Pro | 8B Q4_K_M | 56 t/s | 38–42 t/s |
| Apple M4 Max | 8B Q4_K_M | 111 t/s | 75–85 t/s |
| RTX 4090 | 8B Q4_K_M | 205 t/s | 130–150 t/s |
| RTX 5090 | 8B Q4_K_M | 366 t/s | 240–270 t/s |
| RTX 4090 | 32B Q4_K_M (19.9 GB) | 51 t/s | 33–38 t/s |
| Apple M3 Ultra | 70B Q4_K_M (42.8 GB) | 19 t/s | 12–15 t/s |
| 2 × RTX 3090, layer split | 70B Q4_K_M | 21 t/s | 14–17 t/s |
Three things this table hides. Prefill is far faster than decode on any GPU, so a long prompt costs less wall-clock time than its token count suggests. Splitting a model across two GPUs by layers does not double bandwidth, because the layers run in sequence and one card idles while the other works. And batching changes the economics entirely: serving eight requests at once reuses each weight read across all eight, so aggregate throughput multiplies while each stream feels unchanged. That is the whole reason vLLM exists, and the reason it is wrong for a single user.
CPU vs GPU vs Apple Silicon
An NVIDIA GPU gives the highest bandwidth per dollar and the fastest prompt processing, capped by 24 or 32 GB of VRAM. Apple Silicon gives capacity a consumer GPU cannot touch, at roughly half to a third of the bandwidth. CPU-only works and is the right answer more often than enthusiasts admit, as long as you keep the model small.
NVIDIA is the reference platform. CUDA support lands first, quantization kernels are best optimized there, and a used 3090 remains the value pick for 24 GB. The wall is capacity: a 70B at Q4 needs two cards, which means slots, watts, and cooling for both. The PC building guide covers that side.
Apple Silicon wins on a different axis. Unified memory means the GPU addresses system RAM directly, so a 128 GB MacBook Pro runs a 70B model that no consumer NVIDIA card can hold, silently and on battery. The tradeoff is bandwidth: an M4 Max at 546 GB/s is roughly half a 4090, and prefill lags further because Apple's GPU compute is behind. Long prompts feel slow even when generation is fine.
One Apple-specific trap: macOS caps how much unified memory the GPU may wire down, roughly two-thirds to three-quarters of total RAM. Exceed it and the system swaps, which turns a working setup into an unusable one.
# Check the current limit (0 means "use the default policy")
sysctl iogpu.wired_limit_mb
# Allow 96 GB of a 128 GB machine. Resets on reboot.
sudo sysctl iogpu.wired_limit_mb=98304
# Leave at least 16 GB for macOS itself, or you will trade
# a memory error for a swap storm, which is worse.
CPU-only deserves more respect than it gets. A 4B or 8B model at Q4_K_M on a recent desktop generates faster than most people read and costs nothing extra. Where CPU falls apart is prefill: feeding a 20,000-token document means waiting through work a GPU finishes in about a second. For short prompts and short answers, CPU is fine. For document work, it is not. The same tradeoff appears one tier smaller in edge AI on constrained devices.
Measure Your Own Machine in Five Minutes
Estimates get you a shortlist. Measurement settles it, and llama-bench takes about a minute per configuration.
./build/bin/llama-bench -m models/llama31-8b-Q4_K_M.gguf \
-p 512 -n 128 -ngl 99
# Output columns:
# pp512 prompt processing, 512 tokens -> prefill speed
# tg128 text generation, 128 tokens -> decode speed
# Sweep GPU offload to find the exact cliff on a small card
./build/bin/llama-bench -m models/llama31-8b-Q4_K_M.gguf -ngl 0,16,24,32
# Compare two quants head to head
./build/bin/llama-bench -m a-Q4_K_M.gguf -m b-Q5_K_M.gguf -ngl 99
The offload sweep is the most useful of the three. It shows the exact layer count where throughput falls off a cliff, which is where VRAM ran out. If -ngl 32 is four times faster than -ngl 24 on a 32-layer model, you have found your capacity limit empirically, and you now know how much context you can afford.
The Failure Modes Everyone Hits
Partial offload, silently
Leave two layers on the CPU and those two set the pace for every token. Speed halves, and nothing reports an error.
ollama ps or the llama.cpp load logContext left at the maximum
A 128k default turns a 5 GB model into a 22 GB memory request. Everything that follows looks like a hardware problem and is not.
-c or num_ctx to what you sendDownloading Q8 out of caution
Q8 costs 74 percent more memory and roughly half the speed of Q4_K_M, for a gap almost nobody detects blind.
Wrong chat template
A GGUF with a broken or mismatched template produces rambling that looks like a bad model. It is a formatting bug, not a quality one.
Benchmarking with an empty context
Speed on a two-token prompt says nothing about speed at 30,000 tokens, where attention over the cache starts to dominate.
Laptop thermal throttling
The first sixty seconds are not the steady state. Sustained generation on a thin laptop settles well below its opening pace.
Two more worth naming. Reasoning models spend hundreds of tokens thinking before the visible answer starts, so a 40 tokens-per-second machine can feel stalled; that is the model working, not the runner. And a quantized model hallucinates exactly like the original, so if grounding matters the fix is retrieval, not a higher quant.
When Local Genuinely Beats an API
Work the cost math before the argument. Electricity is the small part; the hardware is the big part, and the deciding variable is how busy you keep it.
Take a 500-watt rig generating 140 tokens per second. An hour of sustained load uses 0.5 kWh, about eight cents at sixteen cents per kWh, and produces roughly 504,000 tokens: about 16 cents per million in power. Now amortize the machine. A 2,500 dollar build used four hours a day for three years is about 4,380 hours, or 57 cents an hour, adding roughly 1.15 dollars per million. Call it 1.30 dollars per million all in, and compare against whatever a hosted model charges today; the LLM price calculator keeps current numbers.
That assumes four busy hours every day for three years. Use the machine twenty minutes a day and the amortized cost triples, which is why price is the weakest argument for going local. The strong arguments are different.
Local Wins Clearly When
- The data cannot leave. Regulated records, client material under confidentiality, or an air-gapped network. No price advantage moves this.
- Volume is high and steady. Classification, extraction, embedding backfills, log summarization: a small model suffices and the hardware never idles.
- You need no network. Field deployments, flights, remote sites, a demo where venue wifi is a risk.
- Rate limits or model changes hurt you. Local weights do not get deprecated, retuned, or throttled at the wrong moment.
- You fine-tuned something. A small model trained on your task can beat a general large one at it, and hosting it yourself is often simpler than uploading it. See the fine-tuning guide.
- Latency matters more than depth. No network round trip at all.
When Not to Run Locally
Skip local in four situations, and skipping is the correct engineering call rather than a concession.
If the task needs frontier reasoning, a 70B at Q4 on your desk is not a substitute for the largest hosted models, and pretending otherwise costs weeks. If your volume is genuinely low, a few thousand tokens a day through an API costs less than the electricity of an idle workstation. If you need very long context, recall the KV arithmetic: 128k tokens exceeds most consumer cards before the weights are counted. And if you are one person shipping a product, the week spent on drivers, quant selection, and offload tuning is a week not spent on the product.
The middle path is to run both. Route bulk, private, or easy work to a local 8B and hard cases to an API. Both endpoints speak the same request shape, so the router is an if-statement, and the structured-output techniques that keep a small model's JSON valid work identically on either side.
Frequently Asked Questions
How much VRAM do I need to run an LLM locally?
Add the weight file, the KV cache, and about 1 GB of overhead. Weights in bytes are parameters times bits-per-weight divided by 8, so an 8B model at Q4_K_M is about 4.9 GB. The KV cache for Llama 3.1 8B at fp16 is 128 KiB per token, so 8,192 tokens adds 1 GiB. Roughly 7 GB total, which fits an 8 GB card. A 70B at Q4_K_M needs about 43 GB of weights plus 2.5 GiB of cache at 8k, meaning two 24 GB cards or a large-memory Apple Silicon machine. The most common mistake is leaving context at the 128k default, which alone demands 16 GiB of cache on an 8B model.
Which GGUF quantization should I use?
Q4_K_M unless you have a specific reason otherwise. It sits near 4.8 bits per weight, cuts a 16-bit model to under a third of its size, and the loss is small enough that most people cannot pick it out blind. Step up to Q5_K_M or Q6_K if the model still fits and you care about code or structured output. Q8_0 is effectively lossless and effectively pointless on consumer hardware, because it halves your speed for a difference you will not see. Below Q3 the damage is obvious: repetition, dropped instructions, broken JSON. A larger model at Q4 usually beats a smaller one at Q8 of the same file size, and that rule stops holding around Q3.
How many tokens per second should I expect from a local LLM?
Generation is bound by memory bandwidth, so divide bandwidth by model file size for a ceiling and expect 60 to 80 percent of it. An RTX 4090 moves 1,008 GB/s, so an 8B at Q4_K_M (4.9 GB) ceilings near 205 tokens per second and lands around 140. An M4 Max at 546 GB/s ceilings near 111 and lands around 75. A desktop CPU on dual-channel DDR5-5600 moves 89.6 GB/s and lands near 10. A 70B at Q4_K_M is about 43 GB, so even the 819 GB/s of an M3 Ultra ceilings at 19. Prompt processing is different: it is compute-bound, and far faster on a GPU.
Is running an LLM locally cheaper than using an API?
Only if you keep the hardware busy. Electricity first: a rig drawing 500 watts for an hour uses 0.5 kWh, about eight cents at sixteen cents per kWh, and at 140 tokens per second produces roughly 500,000 tokens, so power alone is around 16 cents per million. Then amortize the machine: a 2,500 dollar build used four hours a day for three years is about 4,380 hours, roughly 57 cents an hour, adding about 1.15 dollars per million. Near 1.30 dollars per million, local is competitive with mid-tier hosted models and loses badly to the cheapest small ones. Cut usage to twenty minutes a day and that figure triples. The honest reasons to go local are data control, offline operation, and freedom from rate limits, not price.
Method notes: bandwidth figures are manufacturer specifications (NVIDIA GeForce RTX, Apple M-series). Throughput figures are derived as bandwidth divided by file size at a 70 percent efficiency factor, not bench measurements; measure your own with llama-bench. Cost figures use 16 cents per kWh and a three-year amortization, both of which you should replace with your own.