vLLM 2026: Serve LLMs 24× Faster Than Hugging Face Transformers

What Is vLLM?

vLLM is an open-source, high-throughput LLM inference and serving engine. It was created at UC Berkeley's Sky Computing Lab in 2023 and is now maintained by the broader open-source community as the vllm-project on GitHub. Its central technical contribution is PagedAttention — a memory-management algorithm for the attention key-value cache that mirrors operating-system virtual memory paging — combined with continuous batching, which lets new requests join an in-flight batch every decoder step instead of waiting for the slowest sequence to finish.

The practical effect is simple to state: on the same hardware, with the same model, vLLM serves many more concurrent requests per second than vanilla Hugging Face Transformers and earlier serving stacks. The original SOSP 2023 paper by Kwon et al. reported throughput improvements of up to 24x over Hugging Face Transformers and 2–5x over the early TGI baseline. Real-world numbers depend on model size, sequence length distribution, GPU memory, and request concurrency, but the qualitative win — PagedAttention removes the KV-cache memory tax that previously bottlenecked LLM serving — is consistent.

By 2026, vLLM has become the production default for self-hosted LLM serving. It ships with an OpenAI-compatible HTTP server, a Python LLM class for offline batched inference, first-class support for tensor parallelism across multiple GPUs, multiple quantization formats (AWQ, GPTQ, FP8, INT4), prefix caching, speculative decoding, and structured output generation via guided JSON.

Why vLLM in 2026

Three years after release, vLLM occupies the position that TensorFlow Serving once held for traditional ML: the obvious default when you need to put an open-weights LLM behind an HTTP endpoint and serve real traffic. The reasons are concrete.

Throughput. Vanilla transformers.generate() processes one request at a time and pre-allocates a worst-case-sized KV cache for each sequence. PagedAttention plus continuous batching lets vLLM keep the GPU saturated with many in-flight requests, often 5–10x more concurrent sequences for the same model on the same hardware. For any workload with more than a single user, this is the difference between a $4,000/month GPU costing you per-token rates that beat the OpenAI API and one that does not.

OpenAI-compatible API. Run vllm serve meta-llama/Llama-3.1-8B-Instruct and you get an HTTP server on port 8000 that exposes /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models with the exact request and response shapes of the OpenAI API. Existing OpenAI Python or JavaScript clients work by changing the base_url — no other code changes. Self-hosting becomes a configuration switch, not a rewrite.

Multi-GPU support. A single --tensor-parallel-size 8 flag distributes a 70B-parameter model across eight GPUs. NCCL handles the all-reduce communication. There is no manual model sharding code to write.

Quantization support. vLLM loads AWQ-quantized, GPTQ-quantized, and FP8 (Hopper-class GPUs) models directly from Hugging Face with a single --quantization flag. You trade a small quality drop for 2–4x memory reduction and proportionally higher throughput.

Active community. vLLM has thousands of contributors and ships releases roughly monthly. New model architectures — Llama 4, DeepSeek V3, Qwen 3, Mistral Large — are typically supported within days of public release.

PagedAttention Explained

The reason vLLM exists is the KV-cache memory problem — and the reason vLLM dominates self-hosted serving is that PagedAttention solved it cleanly using an idea borrowed from operating systems.

The KV-Cache Memory Problem

During autoregressive generation, every transformer layer caches the keys and values of every preceding token. This cache grows linearly with sequence length and is consulted every decode step. For Llama-3.1-8B at 16-bit precision, the KV cache costs roughly 256KB per token per request. A single request generating 4,000 tokens consumes about 1GB of GPU memory just for the cache — on top of the 16GB of model weights.

Traditional serving systems handled this by pre-allocating a contiguous block of memory equal to the maximum sequence length for every active request. If a request only generated 200 tokens but the limit was 4,000, the other 95% of the cache sat idle and unusable. Internal fragmentation. External fragmentation across many requests of varying lengths made it worse. Real systems achieved roughly 20–40% effective KV-cache memory utilization — the rest was wasted.

The Virtual Memory Analogy

PagedAttention applies the same insight that made operating systems efficient half a century ago. Instead of contiguous allocation, the KV cache is divided into fixed-size blocks (typically 16 tokens each). A logical sequence is a list of pointers to physical blocks — the blocks themselves do not need to be contiguous in GPU memory. Need more cache for a longer sequence? Allocate one more block. Sequence finishes? Free its blocks back to the pool.

The result is near-zero fragmentation. Effective utilization climbs from 20–40% to over 95%. The freed memory is immediately spent on more concurrent sequences — which is where the throughput gain comes from.

Why This Gives 2–4x Throughput

Two compounding effects: (1) more sequences fit in GPU memory simultaneously, so the batch size climbs; (2) continuous batching — vLLM's scheduler can add new requests to an in-flight batch every decoder step rather than waiting for the slowest sequence in the batch to finish. The combination keeps the GPU's matrix-multiply units saturated, which is exactly what makes inference fast.

Installation & First Server

vLLM installs from PyPI in one command on a CUDA-enabled Linux system. Bringing up an OpenAI-compatible server takes one more.

Requirements

Linux (Ubuntu, Debian, RHEL), Python 3.9–3.12, NVIDIA GPU with CUDA 12.1+ and at least 16GB VRAM for an 8B-parameter model in FP16, or 80GB+ for a 70B-parameter model. AMD MI300X, AWS Trainium, and TPU backends are also supported in 2026 builds.

Installation

# Create an isolated environment
python -m venv .venv && source .venv/bin/activate

# Install vLLM (pulls in PyTorch, CUDA bindings, FlashAttention)
pip install vllm

# Verify
python -c "from vllm import LLM; print('vllm ok')"

Launch the OpenAI-Compatible Server

# Single command brings up an OpenAI-compatible server on :8000
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

# Smoke test from another shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role":"user","content":"Hello"}]
  }'

Hit It With the OpenAI Python Client

from openai import OpenAI

# Point the standard OpenAI client at your vLLM server.
# No other code changes — vLLM mirrors the OpenAI request/response shape.
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",  # vLLM does not validate by default
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in one paragraph."}],
    max_tokens=300,
    temperature=0.2,
)
print(response.choices[0].message.content)

Code Sample: Python LLM Class

For offline batched inference — evaluations, dataset labeling, ETL jobs that hit an LLM — the Python LLM class is faster and simpler than HTTP. It loads the model once and processes a list of prompts in a single optimally-batched pass.

from vllm import LLM, SamplingParams

# Load the model once. vLLM downloads from Hugging Face on first run.
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dtype="bfloat16",
    gpu_memory_utilization=0.90,
    max_model_len=8192,
)

# Sampling parameters mirror the OpenAI fields you already know.
sampling = SamplingParams(
    temperature=0.2,
    top_p=0.9,
    max_tokens=256,
)

prompts = [
    "Summarize the SOSP 2023 PagedAttention paper in two sentences.",
    "List five differences between vLLM and Ollama.",
    "Explain tensor parallelism to a junior MLE.",
]

# vLLM batches all prompts into a single optimal pass.
outputs = llm.generate(prompts, sampling)

for out in outputs:
    print("PROMPT:", out.prompt[:60], "...")
    print("COMPLETION:", out.outputs[0].text)
    print("---")

For streaming inside an application, the OpenAI-compatible server with stream=True returns SSE chunks identical to the OpenAI API, and the vllm.AsyncLLMEngine exposes a token-streaming async generator if you are building a custom server.

Multi-GPU & Tensor Parallel

For models that exceed a single GPU's memory — 70B parameters at FP16 needs roughly 140GB, well past any single-GPU budget — vLLM splits the model across GPUs using tensor parallelism.

The Flag

# Spread Llama-3.1-70B across 4 GPUs on one node
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92

Tensor parallelism shards each weight matrix — columns for the input projection, rows for the output projection — so every GPU holds 1/N of every layer. Forward passes do an NCCL all-reduce after each parallelism boundary. The communication is intense; this is the reason tensor-parallel sizes above 8 typically require NVLink or NVSwitch interconnect rather than PCIe.

When Tensor Parallel Helps — and When It Hurts

Tensor parallelism is a memory tool, not a free speedup. If your model already fits on one GPU, adding tensor parallelism usually reduces throughput because of the all-reduce overhead. Only use it when the model itself does not fit. For models that fit on one GPU but need more aggregate throughput, scale horizontally — run multiple single-GPU vLLM replicas behind a load balancer.

For multi-node deployments (e.g. Llama-3.1-405B across two 8xH100 nodes), combine --tensor-parallel-size 8 with --pipeline-parallel-size 2 and an NCCL-aware launcher like Ray.

Quantization (AWQ, GPTQ, FP8)

Quantization compresses model weights from 16-bit floats down to 4-bit or 8-bit integers, cutting memory roughly 4x or 2x respectively. vLLM supports the major quantization formats out of the box and the throughput gain is real — but quality always drops a little. The trade-off is workload-specific.

AWQ (Activation-aware Weight Quantization)

4-bit weight quantization that protects the channels with the largest activations from being quantized aggressively. AWQ models are widely available on Hugging Face (TheBloke and successors). Quality loss on instruction-following benchmarks is typically 1–3 percentage points. Memory drops roughly 4x; throughput rises proportionally because the compute kernel is bandwidth-bound on most GPUs.

GPTQ

Older 4-bit weight-only quantization based on second-order error minimization. Comparable quality to AWQ on most benchmarks; AWQ is slightly faster on modern GPUs because the kernels are better optimized. Choose AWQ for new deployments; GPTQ is appropriate when only GPTQ checkpoints exist.

FP8 (Hopper and Newer)

8-bit floating point natively supported on H100, H200, and B200. Quality loss is near-zero — often within run-to-run noise — while memory drops 2x and throughput rises proportionally. FP8 is the production-default quantization for any deployment on Hopper-class hardware in 2026.

Loading a Quantized Model

# AWQ — 4-bit quantization, ~4x memory reduction
vllm serve "hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4" \
  --quantization awq \
  --tensor-parallel-size 2 \
  --max-model-len 8192

# FP8 on Hopper — near-zero quality loss
vllm serve "meta-llama/Llama-3.1-70B-Instruct" \
  --quantization fp8 \
  --tensor-parallel-size 2

Run your eval suite at full precision and quantized side-by-side before you commit to quantization in production. The "1–3 point" drop is the average; on tasks that depend heavily on long-context reasoning or rare-token vocabulary, the drop can be larger.

vLLM vs TGI vs Ollama vs llama.cpp vs SGLang

vLLM is not the only option, and not the best option for every workload. The honest comparison — with the cases where each alternative wins — is below.

The honest decision tree: vLLM for production self-hosted serving on data-center GPUs — the default, full stop. TGI when you live inside the Hugging Face ecosystem (Inference Endpoints, HF Hub, HF orchestration) and value the deeper integration over raw throughput. Ollama for personal use on a laptop — one-line model swaps and good defaults for non-engineers. llama.cpp when GPU is unavailable — CPU inference, Apple Silicon, edge devices, embedded use. SGLang when your workload is heavily structured (large JSON outputs, complex tool-calling chains, agentic loops with branching) — SGLang's RadixAttention and structured-decoding primitives outperform vLLM in those specific patterns.

Production Patterns

Five patterns separate hobbyist vLLM deployments from production-grade ones.

Kubernetes Deployment

Run vLLM as a standard Deployment with one replica per GPU pod. Use nvidia.com/gpu: 1 in the resource requests for single-GPU models, or a node selector with nvidia.com/gpu: 8 for tensor-parallel-8 models. Mount the Hugging Face cache as a PersistentVolumeClaim so model weights survive pod restarts — cold-starting a 70B model from S3 takes 5–10 minutes; warm-starting from local NVMe takes 30 seconds.

Autoscaling

The standard CPU-utilization HPA does not work for GPU workloads. Use a custom metric — vLLM exposes Prometheus metrics including vllm:num_requests_waiting and vllm:gpu_cache_usage_perc. Scale on queue depth (waiting requests) for latency-sensitive workloads; scale on aggregate token throughput for batch-style workloads. KEDA's Prometheus scaler does this cleanly.

Prefix Caching

If your prompts share a common prefix — system prompts, RAG context blocks, few-shot examples — enable --enable-prefix-caching. vLLM hashes the prefix, caches its KV state across requests, and skips recomputing the shared portion. For agentic workloads with long shared system prompts, this can halve latency and double throughput.

Structured Output with guided_json

vLLM supports constrained decoding via guided_json, guided_regex, and guided_choice in the sampling parameters. Pass a JSON schema, and vLLM will only sample tokens that produce valid JSON matching the schema. This eliminates the "model returned unparseable output" failure mode that plagues unconstrained generation in production. Outlines and XGrammar are the underlying engines depending on the vLLM version.

Observability

Scrape vLLM's /metrics endpoint with Prometheus and dashboard it in Grafana. The metrics that matter: time_to_first_token (TTFT), time_per_output_token (TPOT), num_requests_running, num_requests_waiting, gpu_cache_usage_perc. For request-level tracing, vLLM emits OpenTelemetry spans that integrate with Jaeger or Datadog. Set up alerts on TTFT p99 and queue depth before you have an incident, not after.

Sources: vLLM Documentation, vllm-project/vllm on GitHub, Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023. Throughput figures are from the original paper and vary by workload, hardware, and configuration.