Teaching Models New Behaviors

01. Fine-Tuning vs Prompting — When to Use Which

The most important thing to understand about fine-tuning is that it is not the first tool you should reach for. A large fraction of developers who decide they need to fine-tune have not yet exhausted simpler options. Prompt engineering is free, fast to iterate on, and reversible. RAG injects facts dynamically without training cost. Fine-tuning requires compute, engineering time for data preparation, and produces a static artifact that cannot be easily updated.

The diagnostic question is: is this a knowledge problem or a behavior problem? A knowledge problem means the model does not know certain facts — your internal database, product catalog, or latest regulations. Knowledge problems are almost always better solved by RAG, which injects facts at query time. The model already knows how to reason; you just supply the data.

A behavior problem means the model knows the information but does not act the way you want — wrong tone, wrong format, missing domain conventions, unnecessary caveats. Behavior problems are where fine-tuning shines. It "bakes in" the desired behavior at the weight level, so it persists regardless of what the user says.

What This Means for Practitioners: The Decision Framework

Specific scenarios where fine-tuning wins:

• Consistent output format. If your application requires structured JSON with specific fields, a fine-tuned model just produces it without needing format instructions in every call — saving tokens on every request.
• Specialized vocabulary. Medical, legal, and financial domains have terminology so precise that base models make subtle errors. Fine-tuning on domain examples fixes this.
• Latency and cost at scale. A fine-tuned 7B model can match a prompted GPT-4 on a narrow task at 10-100x lower cost and dramatically lower latency. At millions of daily calls, this arithmetic matters enormously.

When NOT to fine-tune:

• Prompt engineering achieves 85-90% of your goal — the remaining gap rarely justifies the cost.
• Your data distribution changes frequently — fine-tuned models are frozen to their training distribution.
• You have fewer than 100 high-quality examples — you will overfit.

02. LoRA — How It Works and When to Use It

Before LoRA, fine-tuning a 7B model required updating all 7 billion parameters — 60-80 GB of VRAM for gradients and optimizer states, meaning a cluster of expensive GPUs. LoRA (Low-Rank Adaptation) changed this by introducing small additional matrices alongside the existing weights and only training those. The original weights are frozen and never change.

The key insight is that the meaningful updates during fine-tuning have a much simpler structure than the full weight matrices. Even though a weight matrix might be 4096x4096, the actual changes can be captured by two much smaller matrices multiplied together. A LoRA adapter for a 7B model might be 50-200 MB — less than 2% of the model size.

This has a beautiful practical consequence: you can maintain one base model and a library of many LoRA adapters for different tasks. Need a customer service assistant? Swap in the customer service LoRA. Need a code reviewer? Swap in the code review LoRA. One model, many specialists.

LoRA architecture: the frozen pretrained weight W0 (gray, dashed) passes the input through unchanged, while the trained low-rank matrices A and B (cyan) add a small adaptation. Only A and B are updated during training.

What This Means for Practitioners: LoRA Configuration

The key hyperparameters and what they control:

Memory comparison — why LoRA is transformative:

At inference time, you have two options: Merge the adapter into the base model (zero latency overhead — just a normal model) or keep them separate for hot-swapping between adapters without reloading the base model.

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "v_proj", "k_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
)

model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 7,284,015,104 || trainable%: 0.5757

03. QLoRA — Fine-Tuning Models That Do Not Fit in Memory

QLoRA combines LoRA with 4-bit quantization: compress the base model weights to 4 bits per value, reducing memory by 4x, then apply LoRA on top. The base model is stored in 4-bit but computation happens in 16-bit — weights are dequantized on-the-fly during the forward pass, so gradients flow normally through the LoRA parameters.

This is what made it possible to fine-tune a 70B model on a single GPU. Before QLoRA, that required ~140 GB VRAM (two A100s minimum). With QLoRA, the 70B model fits in ~35 GB — a single A100.

What This Means for Practitioners: QLoRA Setup

QLoRA is ~20-30% slower than pure LoRA because of the dequantization overhead. Use LoRA when VRAM allows; use QLoRA when the model would not fit otherwise.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 20,971,520 || all params: 8,051,249,152 || trainable%: 0.2604

04. Alignment — RLHF and DPO

After fine-tuning a model to follow instructions (SFT), you can further align it with human preferences. This is what makes ChatGPT feel helpful rather than producing raw text completions.

RLHF (Reinforcement Learning from Human Feedback) works in three stages: (1) SFT to create a baseline assistant, (2) train a reward model on human preference rankings, (3) use PPO to optimize the model against the reward model. It is effective but complex — three separate models, finicky hyperparameters, and 3-10x the compute cost of SFT.

DPO (Direct Preference Optimization) achieves the same goal in a single training step. Instead of training a separate reward model, DPO directly optimizes on preference pairs — (prompt, chosen response, rejected response) triplets. DPO is now the dominant alignment technique for open-weight models. Nearly all fine-tuned open-source models since mid-2023 use DPO rather than RLHF.

RLHF three-stage pipeline: SFT creates the reference model, the Reward Model learns human preferences from ranked comparisons, and PPO updates the policy to maximize reward while a KL penalty keeps outputs close to the reference model.

What This Means for Practitioners: Choosing an Alignment Method

DPO dataset format is straightforward:

# One example per line in JSONL
{
  "prompt": "Explain why the sky is blue.",
  "chosen": "The sky appears blue due to Rayleigh scattering...",
  "rejected": "Because molecules scatter short wavelengths, blue."
}

from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

dpo_config = DPOConfig(
    beta=0.1,
    learning_rate=5e-7,
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    output_dir="./dpo_output",
)

dataset = load_dataset("your_org/preference_dataset")

trainer = DPOTrainer(
    model=policy_model,
    ref_model=reference_model,
    args=dpo_config,
    train_dataset=dataset["train"],
    processing_class=tokenizer,
)
trainer.train()

05. Dataset Preparation

In fine-tuning, data quality is the single most important variable. You can choose the perfect hyperparameters and training infrastructure — all worthless if your data is noisy, inconsistent, or misaligned with what you want.

A counterintuitive but well-validated finding: 100 perfectly curated examples consistently outperform 10,000 mediocre ones for instruction tuning. The model is already highly capable — it just needs to be steered.

What This Means for Practitioners: Data Preparation Checklist

Dataset size guidelines:

The standard format is JSONL with a messages array:

{
  "messages": [
    {"role": "system", "content": "You are a medical documentation assistant..."},
    {"role": "user", "content": "Summarize the following clinical note..."},
    {"role": "assistant", "content": "Chief Complaint: chest pain..."}
  ]
}

Data quality checklist (do not skip these):

• Deduplication — Remove near-identical examples. Duplicates cause overfitting to specific phrasings.
• Quality filtering — Review every example manually for datasets under 1,000. Use LLM-as-judge for larger sets.
• PII removal — Scrub names, addresses, phone numbers, emails. Essential for compliance.
• Format validation — Every example must be syntactically valid with expected fields.
• Distribution matching — Training data must match real production inputs. If users send terse queries, do not train on elaborate prompts.
• Separate test set from real data — Never evaluate on synthetic data from the same generation process as training data.

Synthetic data generation using a strong model (GPT-4o, Claude) to create training data for a weaker model is increasingly standard. This is the approach behind Microsoft's Phi family.

from openai import OpenAI
import json

client = OpenAI()

def generate_training_example(seed_topic: str) -> dict:
    meta_prompt = f"""Generate a realistic user question about {seed_topic}
and a high-quality expert answer.
Return JSON: {{"prompt": "...", "response": "..."}}"""

    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": meta_prompt}],
        response_format={"type": "json_object"},
    )
    data = json.loads(result.choices[0].message.content)
    return {
        "messages": [
            {"role": "user", "content": data["prompt"]},
            {"role": "assistant", "content": data["response"]},
        ]
    }

seed_topics = ["drug interactions", "dosage calculations", "clinical terminology"]
examples = [generate_training_example(t) for t in seed_topics * 100]

with open("train.jsonl", "w") as f:
    for ex in examples:
        f.write(json.dumps(ex) + "\n")

06. OpenAI Fine-Tuning API

OpenAI provides a fully managed fine-tuning service: upload a JSONL file, submit a job, and receive a custom model endpoint. No GPU to provision, no training loop to write. The fine-tuned model is accessed through the standard Chat Completions API — the only difference is the model ID.

What This Means for Practitioners: The Complete Workflow

Step 1 — Upload data:

from openai import OpenAI
client = OpenAI()

with open("train.jsonl", "rb") as f:
    train_file = client.files.create(file=f, purpose="fine-tune")
with open("val.jsonl", "rb") as f:
    val_file = client.files.create(file=f, purpose="fine-tune")

Step 2 — Create fine-tuning job:

job = client.fine_tuning.jobs.create(
    training_file=train_file.id,
    validation_file=val_file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={"n_epochs": "auto", "learning_rate_multiplier": "auto", "batch_size": "auto"},
    suffix="medical-notes-v1",
)

Step 3 — Monitor and use:

import time
while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    if job.status in ["succeeded", "failed", "cancelled"]:
        break
    time.sleep(60)

# Use the fine-tuned model — no format instructions needed
response = client.chat.completions.create(
    model=job.fine_tuned_model,  # ft:gpt-4o-mini-2024-07-18:your-org:medical-notes-v1:AbCdEf12
    messages=[{"role": "user", "content": "Summarize this clinical note: ..."}],
    temperature=0.2,
)

OpenAI vs. self-hosted fine-tuning decision:

Interview Ready

How to Explain This in 2 Minutes

Likely Interview Questions

Model Answers

Common Mistakes

• Fine-tuning for knowledge instead of behavior. Using fine-tuning to teach facts when RAG would be cheaper, more accurate, and easier to update. Fine-tuning bakes information into weights where it cannot be cited or refreshed.
• Training on low-quality data. 500 carefully curated examples outperform 5,000 scraped, unchecked ones. Always inspect data manually.
• Ignoring the validation loss curve. Without a validation split, you cannot detect overfitting. Many practitioners train too many epochs and ship models that memorize rather than generalize.

Previous Module

03 · APIs for LLMs

Next Module

05 · LLM Hosting

Next: Deploying Models