vLLM: High-Throughput LLM Serving with PagedAttention

Installation & Quick Start

vLLM requires an NVIDIA GPU with CUDA 11.8+ or 12.x. It downloads models from HuggingFace automatically. For gated models (Llama, Gemma), set your HuggingFace token first.

# Install vLLM (Python 3.9+, NVIDIA GPU required)
pip install vllm

# For specific CUDA versions:
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu121

# Set HuggingFace token for gated models (Llama, Gemma, etc.)
export HF_TOKEN=hf_your_token_here
huggingface-cli login  # or set HF_TOKEN env var

# Start the OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.2-8B-Instruct \
  --port 8000 \
  --host 0.0.0.0

# Test the server
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.2-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

PagedAttention & Continuous Batching

Understanding vLLM's two core innovations helps you configure it correctly and understand its throughput claims:

# PagedAttention — GPU memory management
# ─────────────────────────────────────────────────────────────
# Without PagedAttention (naive):
#   Request A: [████████████████████████____] ← 25% wasted (pre-allocated)
#   Request B: [████████________] ← 50% wasted
#   GPU memory utilization: ~40-60%
#
# With PagedAttention:
#   KV cache divided into 16-token "pages"
#   Pages allocated on demand, can be non-contiguous
#   GPU memory utilization: ~95%+
#   Shared prefix caching: requests with same system prompt
#   share KV cache pages → further memory savings

# Continuous batching — request scheduling
# ─────────────────────────────────────────────────────────────
# Without (static batching):
#   Batch waits for ALL requests to finish before starting new ones
#   Short requests waste time waiting for long ones
#
# With continuous batching:
#   New requests inserted into the batch at every iteration step
#   Finished sequences immediately replaced with new ones
#   Result: much higher GPU utilization under real traffic

# Configure GPU memory and concurrency:
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --gpu-memory-utilization 0.90 \   # Use 90% of GPU memory for KV cache
  --max-num-seqs 256 \              # Max concurrent sequences
  --max-model-len 32768             # Max context length

Quantization & Multi-GPU Serving

vLLM supports multiple quantization methods and scales horizontally across GPUs using tensor parallelism:

# Quantization — reduce VRAM requirements
# AWQ 4-bit (recommended for quality/speed balance)
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-2-70B-Chat-AWQ \
  --quantization awq \
  --dtype float16

# GPTQ 4-bit
python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
  --quantization gptq

# FP8 (NVIDIA H100/H200/A100 only — fastest)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --quantization fp8

# Multi-GPU tensor parallelism (single node)
# Llama 70B across 4x A100-40GB
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.95

# Check GPU memory requirements:
# 7B FP16: ~14 GB  | 7B AWQ-4bit: ~4 GB
# 13B FP16: ~26 GB | 13B AWQ: ~7 GB
# 70B FP16: ~140 GB | 70B AWQ: ~35 GB

vLLM vs TGI vs TensorRT-LLM