LLM Fine-tuning Complete Guide: When to Fine-tune vs RAG vs Prompting (2026)

Fine-tuning is one of the most misunderstood capabilities in the LLM ecosystem. Developers reach for it too early, apply it to problems better solved by prompt engineering, and then wonder why results are worse than expected.

This guide gives you an honest decision framework: when to use prompt engineering, when to use RAG, and when fine-tuning is actually the right tool. Then we go deep on the technical execution — data preparation, PEFT/LoRA, QLoRA on consumer hardware, OpenAI’s fine-tuning API, and how to measure whether it worked.

The Core Decision Framework

Before touching a training script, answer these three questions:

1. Is it a knowledge problem or a behavior problem?

Knowledge problem: The model doesn’t know a fact, document, or piece of information.

• Solution: RAG (Retrieval-Augmented Generation)

Behavior problem: The model knows how to do something but doesn’t do it in the style, format, or reasoning pattern you need.

• Solution: Fine-tuning

Fine-tuning doesn’t teach models facts reliably. It teaches them patterns of behavior. If you’re trying to inject proprietary knowledge (your docs, your data), RAG will almost always outperform fine-tuning and be far cheaper to maintain.

2. Can prompt engineering solve it?

Before committing to fine-tuning, try:

• System prompt engineering: Define role, output format, examples, constraints
• Few-shot examples: Include 3-10 worked examples in the prompt
• Chain-of-thought prompting: Ask the model to reason step by step
• Structured output enforcement: Use JSON mode, grammar-constrained generation, or tool-calling

Modern frontier models (GPT-4o, Claude Sonnet, Gemini 1.5 Pro) respond extremely well to carefully crafted prompts. If a 500-token system prompt achieves 90% of your target quality, the cost and complexity of fine-tuning is rarely justified.

3. Do you have enough quality labeled data?

Fine-tuning requirements by approach:

Method	Minimum Examples	Sweet Spot	Notes
OpenAI fine-tuning	50-100	500-2000	Quality >> quantity
LoRA (7B model)	500	2000-10000	Needs careful data prep
QLoRA (7B model)	500	2000-10000	Same as LoRA
Full fine-tune (7B)	5000+	50000+	Requires serious GPU
Full fine-tune (70B+)	10000+	100000+	Multi-GPU cluster required

If you have fewer than 500 examples, start with prompt engineering and few-shot examples.

---

When Fine-tuning Actually Makes Sense

Fine-tuning is the right choice when:

Consistent output format is critical. If every response must follow a specific JSON schema, markdown structure, or domain-specific syntax, fine-tuning is far more reliable than prompt engineering — especially for edge cases.

Latency and cost matter at scale. A fine-tuned 7B or 13B model can match GPT-4 quality on narrow tasks while costing 95% less per token and running with 3x lower latency. For high-volume production APIs, this arithmetic matters.

Domain-specific reasoning patterns. Medical diagnosis reasoning, legal clause analysis, code review in a specific style — these behavioral patterns can be baked into a smaller model through fine-tuning, resulting in a specialist that outperforms generalists on the specific task.

Reducing prompt length at scale. If your system prompt is 2000+ tokens of instructions and examples, fine-tuning can internalize those patterns and reduce your per-request cost significantly.

Privacy requirements prevent using frontier APIs. If your data can’t leave your infrastructure, fine-tuning an open-weight model (Llama 3, Mistral, Qwen) on your own hardware is the path to LLM capabilities without data exposure.

When Fine-tuning Does NOT Make Sense

• You want the model to “know more” (use RAG)
• You have fewer than 200 high-quality examples
• Your task changes frequently (prompt engineering is cheaper to update)
• You’re in early exploration / prototyping phase
• Your eval set doesn’t clearly show that a base model fails on your task

---

Step 1: Preparing Training Data

Data quality is the single most important factor in fine-tuning success. Bad data produces bad models, and more bad data makes it worse.

The Format: Instruction-Response Pairs

Most fine-tuning today uses the instruction-following format (also called supervised fine-tuning / SFT):

[
  {
    "messages": [
      {"role": "system", "content": "You are a SQL query optimizer. Return only the optimized query and a brief explanation."},
      {"role": "user", "content": "SELECT * FROM orders WHERE customer_id = 123 AND status = 'pending' ORDER BY created_at DESC"},
      {"role": "assistant", "content": "```sql\nSELECT id, customer_id, status, created_at, total_amount\nFROM orders\nWHERE customer_id = 123 AND status = 'pending'\nORDER BY created_at DESC\nLIMIT 100;\n```\n\nAdded explicit column selection to reduce I/O and added LIMIT to prevent full table scans on unbounded queries."}
    ]
  }
]

Data Collection Strategies

Human-generated gold examples: Most expensive but highest quality. Domain experts write ideal responses. Even 200 carefully crafted examples can produce a good LoRA adapter.

GPT-4 / Claude distillation: Generate synthetic training data using a frontier model, then human-review a sample. Use prompts like: “Given this input, write the ideal response following these guidelines: [your criteria].” Filter outputs carefully.

Existing logs + relabeling: If you have historical human-written outputs (support tickets, code reviews, clinical notes), clean and reformat them. Watch for distribution issues — historical data may reflect past behavior you’re trying to change.

Data Quality Checklist

• All responses follow your target format exactly
• No examples with hallucinated facts
• Consistent tone and style across examples
• Edge cases and failure modes are represented
• Training and validation splits are disjoint (no data leakage)
• Reviewed a random sample manually before training

Splitting Your Dataset

import json
import random

with open("training_data.jsonl") as f:
    data = [json.loads(line) for line in f]

random.shuffle(data)
split_idx = int(len(data) * 0.9)

train_data = data[:split_idx]
val_data = data[split_idx:]

with open("train.jsonl", "w") as f:
    for item in train_data:
        f.write(json.dumps(item) + "\n")

with open("val.jsonl", "w") as f:
    for item in val_data:
        f.write(json.dumps(item) + "\n")

print(f"Train: {len(train_data)} | Val: {len(val_data)}")

---

Step 2: PEFT and LoRA Explained

Full fine-tuning updates every parameter in the model — for a 7B parameter model, that’s 28GB of gradients and optimizer states, requiring 80GB+ of GPU VRAM. This is impractical for most teams.

Parameter-Efficient Fine-Tuning (PEFT) is the solution. Instead of updating all weights, PEFT methods inject a small number of trainable parameters into the frozen model. The most popular approach is LoRA (Low-Rank Adaptation).

How LoRA Works

For each transformer weight matrix W, LoRA adds two small matrices A and B where rank r << original dimension:

W_new = W_frozen + (A × B) × scaling_factor

A typical 7B model has ~250M trainable parameters in a full fine-tune. A LoRA adapter with rank 16 has ~8M trainable parameters — about 3% of the model. Training is 4-8x faster, VRAM usage drops dramatically, and you can store dozens of task-specific adapters without storing duplicate full models.

QLoRA: LoRA + Quantization

QLoRA (Quantized LoRA) takes this further by loading the base model in 4-bit quantization (using bitsandbytes), reducing VRAM by another 4x. A 7B model that needed 28GB in full fine-tune now fits in 12GB of VRAM for QLoRA training.

This means you can fine-tune a 7B model on a single RTX 4090 (24GB VRAM) or a cloud A10G.

---

Step 3: Fine-tuning with Hugging Face + PEFT

Installation

pip install transformers peft datasets accelerate bitsandbytes trl

Full QLoRA Training Script

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# ── Config ──────────────────────────────────────────────────────────────
BASE_MODEL = "meta-llama/Meta-Llama-3-8B-Instruct"
OUTPUT_DIR = "./llama3-finetuned"
DATASET_FILE = "train.jsonl"

# QLoRA: load model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

# ── Load model and tokenizer ─────────────────────────────────────────────
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model = prepare_model_for_kbit_training(model)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# ── LoRA configuration ───────────────────────────────────────────────────
lora_config = LoraConfig(
    r=16,                          # rank — higher = more capacity, more params
    lora_alpha=32,                 # scaling factor (usually 2x rank)
    target_modules=[               # which attention layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 8,388,608 || all params: 8,039,137,280 || trainable%: 0.10

# ── Dataset ──────────────────────────────────────────────────────────────
dataset = load_dataset("json", data_files={"train": DATASET_FILE}, split="train")

def format_chat(example):
    """Format messages list into a single training string."""
    messages = example["messages"]
    formatted = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=False
    )
    return {"text": formatted}

dataset = dataset.map(format_chat)

# ── Training arguments ───────────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,   # effective batch size = 8
    optim="paged_adamw_32bit",
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=10,
    save_strategy="epoch",
    fp16=True,
    report_to="none",               # set to "wandb" for experiment tracking
)

# ── Trainer ──────────────────────────────────────────────────────────────
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=lora_config,
    dataset_text_field="text",
    max_seq_length=2048,
    tokenizer=tokenizer,
    args=training_args,
)

trainer.train()
trainer.save_model(OUTPUT_DIR)
print(f"Model saved to {OUTPUT_DIR}")

Loading and Inferring from a LoRA Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "./llama3-finetuned")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

messages = [
    {"role": "system", "content": "You are a SQL query optimizer."},
    {"role": "user", "content": "SELECT * FROM users WHERE email = '[email protected]'"}
]

input_ids = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True
).to("cuda")

outputs = model.generate(input_ids, max_new_tokens=512, temperature=0.1)
response = tokenizer.decode(outputs[0][input_ids.shape[1]:], skip_special_tokens=True)
print(response)

---

Step 4: OpenAI Fine-tuning API

If you want managed fine-tuning without touching GPU infrastructure, OpenAI’s API fine-tunes GPT-4o-mini (and GPT-3.5-turbo) with minimal code.

Pricing (2026)

• Training: $0.003 per 1K tokens (GPT-4o-mini), $0.008 per 1K tokens (GPT-4o)
• Inference on fine-tuned model: 2-4x the base model’s inference price
• Storage: Free for up to 10 fine-tuned models

A typical run of 1,000 examples averaging 500 tokens each = 500K training tokens = ~$1.50 for GPT-4o-mini. Affordable enough to iterate quickly.

Full OpenAI Fine-tuning Workflow

from openai import OpenAI
import time

client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

# ── Step 1: Upload training file ─────────────────────────────────────────
with open("train.jsonl", "rb") as f:
    upload_response = client.files.create(file=f, purpose="fine-tune")

file_id = upload_response.id
print(f"Uploaded file: {file_id}")

# ── Step 2: Create fine-tuning job ───────────────────────────────────────
job = client.fine_tuning.jobs.create(
    training_file=file_id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={
        "n_epochs": 3,             # 3 is usually optimal; more risks overfitting
        "batch_size": "auto",
        "learning_rate_multiplier": "auto"
    },
    suffix="my-task-v1"            # becomes part of the model name
)

job_id = job.id
print(f"Fine-tuning job started: {job_id}")

# ── Step 3: Poll for completion ──────────────────────────────────────────
while True:
    job_status = client.fine_tuning.jobs.retrieve(job_id)
    status = job_status.status
    print(f"Status: {status}")

    if status in ("succeeded", "failed", "cancelled"):
        break
    time.sleep(60)

if status == "succeeded":
    fine_tuned_model = job_status.fine_tuned_model
    print(f"Fine-tuned model ready: {fine_tuned_model}")
else:
    print(f"Job failed: {job_status.error}")

# ── Step 4: Use the fine-tuned model ────────────────────────────────────
response = client.chat.completions.create(
    model=fine_tuned_model,  # e.g. "ft:gpt-4o-mini-2024-07-18:my-org:my-task-v1:abc123"
    messages=[
        {"role": "system", "content": "You are a SQL query optimizer."},
        {"role": "user", "content": "SELECT * FROM orders WHERE status = 'pending'"}
    ],
    temperature=0.1
)
print(response.choices[0].message.content)

---

Step 5: Cost Estimates

Self-Hosted Fine-tuning (GPU Cloud)

Model Size	Method	GPU	Training Time (1K examples)	Approx Cost
7B	QLoRA	1x A10G (24GB)	~2-4 hours	$3-8
7B	QLoRA	1x A100 (80GB)	~1-2 hours	$3-6
13B	QLoRA	1x A100 (80GB)	~3-5 hours	$9-15
70B	QLoRA	4x A100 (80GB)	~8-16 hours	$96-192

Providers: Lambda Labs, RunPod, Vast.ai (cheapest), Modal, Google Colab Pro+ (for small experiments).

Managed Fine-tuning

Provider	Model	~1K examples cost	Inference markup
OpenAI	GPT-4o-mini	$1.50	2x base price
OpenAI	GPT-4o	$4.00	2x base price
Together AI	Llama 3 8B	$0.50-2	Low inference cost
Replicate	Various	Per-second compute	Pay per inference

---

Step 6: Evaluation — Did It Work?

This is where most fine-tuning guides fall short. Training loss going down is not success.

Define Your Evaluation Criteria First

Before training, build a held-out evaluation set of 50-200 examples the model will never see during training. For each example, define what “correct” means:

• Exact match: The output must match a target string exactly (good for structured outputs, code generation)
• LLM-as-judge: Use GPT-4 or Claude to score outputs 1-5 against your criteria
• Task-specific metrics: F1 score for classification, ROUGE for summarization, pass@k for code
• Human evaluation: The gold standard, but expensive

Automated Evaluation Script

from openai import OpenAI
import json

client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

FINE_TUNED_MODEL = "ft:gpt-4o-mini-2024-07-18:my-org:my-task-v1:abc123"
BASE_MODEL = "gpt-4o-mini-2024-07-18"

def evaluate_model(model: str, eval_data: list, judge_model: str = "gpt-4o") -> dict:
    scores = []

    for example in eval_data:
        # Get model response
        response = client.chat.completions.create(
            model=model,
            messages=example["messages"][:-1],  # exclude the target assistant turn
            temperature=0.0
        )
        prediction = response.choices[0].message.content
        target = example["messages"][-1]["content"]

        # LLM-as-judge scoring
        judge_response = client.chat.completions.create(
            model=judge_model,
            messages=[{
                "role": "user",
                "content": f"""Score this model output from 1-5 based on how well it matches the target.

Target: {target}

Model output: {prediction}

Return only a JSON object: {{"score": <1-5>, "reason": "<brief explanation>"}}"""
            }],
            response_format={"type": "json_object"},
            temperature=0.0
        )
        result = json.loads(judge_response.choices[0].message.content)
        scores.append(result["score"])

    return {
        "model": model,
        "avg_score": sum(scores) / len(scores),
        "scores": scores,
        "n_examples": len(scores)
    }

# Load evaluation set
with open("val.jsonl") as f:
    eval_data = [json.loads(line) for line in f][:50]  # use first 50

# Compare base vs fine-tuned
base_results = evaluate_model(BASE_MODEL, eval_data)
ft_results = evaluate_model(FINE_TUNED_MODEL, eval_data)

print(f"Base model avg score: {base_results['avg_score']:.2f}/5")
print(f"Fine-tuned model avg score: {ft_results['avg_score']:.2f}/5")
print(f"Improvement: {ft_results['avg_score'] - base_results['avg_score']:+.2f}")

Signs Fine-tuning Worked

• Average score improves by 0.5+ on your rubric
• The model consistently follows your output format without reminders
• Fewer refusals or off-topic responses on your specific use case
• Qualitative examples look noticeably better to domain experts

Signs Fine-tuning Did NOT Work

• Val loss is lower than train loss (overfitting — need more data or fewer epochs)
• Model performs better on training examples but not on new inputs
• The model “forgets” general capabilities (catastrophic forgetting — use lower learning rate)
• Scores on your task improved but general benchmarks dropped significantly

---

The Full Decision Flowchart

Is this a knowledge/factual retrieval problem?
  └─ YES → Use RAG (Retrieval-Augmented Generation)
  └─ NO ↓

Can a carefully crafted prompt achieve 80%+ of target quality?
  └─ YES → Use prompt engineering + few-shot examples
  └─ NO ↓

Do you have 500+ quality labeled examples?
  └─ NO → Collect more data first; use prompting in the meantime
  └─ YES ↓

Is data privacy a hard constraint?
  └─ YES → Self-hosted fine-tuning (QLoRA on open-weight model)
  └─ NO ↓

Do you need best possible quality on a narrow task at scale?
  └─ YES → Fine-tune (OpenAI API or self-hosted)
  └─ NO → Prompt engineering is probably good enough

---

Summary

Fine-tuning is a powerful tool but an expensive one to wield poorly. The most common mistake is reaching for it before exhausting simpler options.

The right mental model: prompt engineering and RAG are free iterations, fine-tuning is a paid commitment. Use the cheap options to define exactly what quality you’re targeting, build your eval set, and only then invest in fine-tuning if there’s a clear gap.

When you do fine-tune:

• Use QLoRA for open-weight models — it makes 7B-13B accessible on single GPUs
• Use OpenAI’s API for managed simplicity on narrow tasks
• Evaluate rigorously with a held-out set and LLM-as-judge scoring
• Treat your training data with the same care you’d give production code

For related developer tools while building AI pipelines, check out DevPlaybook’s JSON Formatter for validating JSONL training files and the Diff Checker for comparing model outputs side by side.