llama.cpp: Efficient Local LLM Inference on CPU & GPU

Building llama.cpp

llama.cpp is a C++ project built with CMake. Choose the right build flags for your hardware:

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# CPU-only build (universal, works everywhere)
cmake -B build
cmake --build build --config Release -j$(nproc)

# NVIDIA GPU (CUDA) — requires CUDA toolkit
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Apple Silicon (Metal) — best macOS performance
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release

# AMD GPU (ROCm/HIP) — Linux only
cmake -B build -DGGML_HIPBLAS=ON
cmake --build build --config Release

# Vulkan — cross-platform GPU (Intel, AMD, NVIDIA)
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release

# Binaries are in build/bin/
# Key binaries: llama-cli, llama-server, llama-embedding

GGUF Quantization: Choosing the Right Level

GGUF quantization compresses model weights to reduce file size and RAM usage. Here's a practical guide to choosing the right quantization for your hardware and quality requirements:

# Quantization comparison for Llama 3.2 7B:
# ──────────────────────────────────────────────────────────────
# Format     Size    RAM     Quality   Speed     Recommended
# Q2_K       2.7 GB  ~5 GB   ★★☆☆☆    Fastest   Low RAM only
# Q3_K_M     3.3 GB  ~6 GB   ★★★☆☆    Fast      Constrained RAM
# Q4_0       3.8 GB  ~6 GB   ★★★☆☆    Fast      Legacy, prefer Q4_K_M
# Q4_K_M     4.1 GB  ~7 GB   ★★★★☆    Fast      ← BEST BALANCE
# Q4_K_S     3.9 GB  ~7 GB   ★★★★☆    Fast      Slightly smaller than _M
# Q5_K_M     4.8 GB  ~8 GB   ★★★★☆    Moderate  High quality
# Q5_K_S     4.7 GB  ~8 GB   ★★★★☆    Moderate  Good for coding
# Q6_K       5.5 GB  ~8 GB   ★★★★★    Moderate  Near-lossless
# Q8_0       7.7 GB  ~10 GB  ★★★★★    Moderate  Near-lossless
# F16        14.1 GB ~16 GB  ★★★★★    Slowest   Reference quality

# Quantize your own model from HuggingFace:
# Step 1: Convert to GGUF
python convert_hf_to_gguf.py /path/to/hf/model --outtype f16

# Step 2: Quantize
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

Server Mode (OpenAI-Compatible API)

The llama-server binary starts an HTTP server with an OpenAI-compatible API and a built-in chat UI:

# Start the server with GPU offloading
./build/bin/llama-server \
  --model models/llama-3.2-7b-instruct-q4_k_m.gguf \
  --port 8080 \
  --host 0.0.0.0 \
  --ctx-size 8192 \
  --n-gpu-layers 35 \    # Offload 35 layers to GPU (set 999 for all)
  --threads 8 \          # CPU threads for CPU layers
  --batch-size 512 \     # Prompt processing batch size
  --parallel 2           # Number of parallel request slots

# Chat UI: http://localhost:8080
# OpenAI API: http://localhost:8080/v1/chat/completions

# Use with Python openai SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="llamacpp")
resp = client.chat.completions.create(
    model="llama-3.2-7b",
    messages=[{"role": "user", "content": "Explain PagedAttention"}]
)
print(resp.choices[0].message.content)

Python Bindings: llama-cpp-python

llama-cpp-python wraps the C++ library for direct use in Python scripts. It also includes a built-in OpenAI-compatible server.

# Install (CPU)
pip install llama-cpp-python

# Install with CUDA support
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

# Install with Metal support (macOS)
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

# Basic usage
from llama_cpp import Llama

llm = Llama(
    model_path="./models/llama-3.2-7b-q4_k_m.gguf",
    n_ctx=8192,          # Context window
    n_gpu_layers=35,     # GPU layers (0 = CPU only, -1 = all)
    n_threads=8,         # CPU threads
    verbose=False
)

# Chat completion
response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a Python decorator for timing functions"}
    ],
    max_tokens=512,
    temperature=0.7
)
print(response["choices"][0]["message"]["content"])

# Start built-in OpenAI server
python -m llama_cpp.server --model model.gguf --port 8000

llama.cpp vs Ollama