KoboldCpp: Local LLM Inference for Creative Writing & Roleplay

Installation & Quick Start

KoboldCpp is distributed as a single executable file — no installation required. Download the right binary for your platform from GitHub Releases:

# Download from: https://github.com/LostRuins/koboldcpp/releases

# Windows (NVIDIA GPU)
# Download: koboldcpp_cu12.exe (CUDA 12) or koboldcpp_cu11.exe (CUDA 11)

# Windows (CPU only)
# Download: koboldcpp.exe

# macOS (Apple Silicon)
# Download: koboldcpp-mac-arm64

# Linux (CPU + CUDA)
# Download: koboldcpp-linux-x64 or koboldcpp-linux-x64-cuda1150

# Run with the GUI launcher (recommended for beginners):
./koboldcpp.exe  # Opens a configuration GUI

# Run headless with command line:
./koboldcpp.exe \
  --model /path/to/model.gguf \
  --contextsize 8192 \
  --gpulayers 35 \
  --port 5001

# Access the web UI at http://localhost:5001
# KoboldAI API at http://localhost:5001/api/v1
# OpenAI API at http://localhost:5001/v1

World Info, Memory & Author's Note

These three features are what make KoboldCpp uniquely powerful for long-form creative writing and roleplay:

# MEMORY (always-on context block)
# Text always included at the start of the model's context
# Use for: setting, rules, character descriptions, writing style notes
# Example Memory entry:
# "Setting: A cyberpunk city in 2157. Technology is advanced but society
#  is stratified. The protagonist is a hacker named Yara, 28, cynical
#  but idealistic. Writing style: noir, present tense, first person."

# AUTHOR'S NOTE (injected near the end of context)
# Guides the model's immediate output style
# Example: "[Write in the style of Philip K. Dick. Build tension slowly.]"
# Position: 3 (default) = inject 3 turns from the bottom of context

# WORLD INFO (keyword-triggered dynamic injection)
# Each entry has: title, content, trigger keywords, priority
# Example entries:
# Title: Yara's Apartment
# Content: "Yara lives in a cramped 200 sq ft pod in sector 7.
#            Her walls are covered in hacked display panels."
# Triggers: "apartment, home, pod, sector 7"
# → Injected automatically when these words appear in conversation

# Title: Nexus Corp
# Content: "Nexus Corp is the megacorporation controlling water and power.
#            CEO: Director Chen. Known for human experimentation."
# Triggers: "nexus, corporation, nexus corp, director chen"

SillyTavern Integration

SillyTavern is the most popular character-based roleplay frontend. It connects to KoboldCpp's API to handle all inference while SillyTavern manages characters, personas, and UI:

# 1. Start KoboldCpp with your model:
./koboldcpp.exe --model mymodel.gguf --contextsize 8192 --gpulayers 35

# 2. Install SillyTavern:
git clone https://github.com/SillyTavern/SillyTavern
cd SillyTavern
npm install
node server.js

# 3. Connect SillyTavern to KoboldCpp:
# Open SillyTavern at http://localhost:8000
# Click the API icon (top bar)
# Select "KoboldAI" as the API type
# Enter: http://localhost:5001
# Click "Test Connection" — should show green

# 4. Key SillyTavern features with KoboldCpp:
# - Character cards (PNG with embedded JSON)
# - Persona management (your character)
# - World info editor (syncs with KoboldCpp)
# - Lorebooks (additional world info)
# - Prompt templates (Alpaca, ChatML, Llama3, etc.)
# - Chat export and import

# Character cards: https://chub.ai (largest library)

KoboldCpp vs text-generation-webui