LLMs, SLMs & Multimodal Models

01. The LLM Landscape

Model landscape positioned by capability (vertical) versus cost-per-token (horizontal). The dashed curve represents the value frontier — models on it deliver optimal capability for their price point.

When people say a model is "large," they are referring to its parameter count — the number of learned numerical weights that encode the model's knowledge and behavior. But parameter count is just the headline number. What actually determines a model's capability is the combination of parameters, training data quality, and compute used during training. A model with fewer parameters trained on better, more carefully curated data can outperform a larger model trained sloppily. This is why some of the most surprising results in recent years have been small models punching far above their weight.

The major providers break into two camps. On the closed-weight side: OpenAI (GPT-4o, o1, o3), Anthropic (Claude 3.5 and 3.7), Google DeepMind (Gemini Flash, Pro, Ultra), and others. These companies keep weights proprietary and give you access through paid APIs. On the open-weight side: Meta AI (Llama 3.x), Mistral AI, Alibaba (Qwen 2.5), Microsoft (Phi-3, Phi-4), Google (Gemma 2), and DeepSeek. These labs release model weights publicly — you can download them, run them locally, fine-tune them, and deploy on your own infrastructure.

The landscape changes fast. Models that were state-of-the-art in early 2024 were beaten by mid-2024. GPT-3.5-level capability, which cost meaningful money in 2022, can now run locally for free on a mid-range laptop. The specific model names you memorize today will be outdated within months, but the frameworks for evaluating and selecting models stay relevant.

What This Means for Practitioners

Closed-weight vs. open-weight is an engineering decision, not a philosophical one. Here is when to choose each:

Context windows have expanded dramatically but longer is not always better. Claude 3.7 Sonnet offers 200K tokens, Gemini 2.0 can handle 1M tokens. But research consistently shows a "lost in the middle" problem — information buried deep in a long context is recalled less reliably. Structured retrieval (RAG) often outperforms naive context stuffing even when the window is technically large enough.

Mixture-of-Experts (MoE) is how frontier models stay fast despite being huge. An MoE model has multiple feed-forward blocks ("experts") and a routing network that selects a small subset for each token. Mixtral 8x7B has 46B total parameters but only activates about 12B per inference step — quality closer to a 40B+ dense model at the speed of a 12B model. GPT-4 is widely speculated to use this approach.

Benchmarks are useful for rough comparisons but unreliable for your specific task. MMLU measures factual recall, not the reasoning or instruction-following that matters in production. The benchmarks that matter most: LMSYS Chatbot Arena (human preference voting — the gold standard), LiveBench (dynamically updated to prevent contamination), and GPQA (graduate-level questions that even PhD experts struggle with). Always build your own evaluation set for your specific use case.

02. Key Models — What You Need to Know

Rather than memorize every model's specifications, focus on understanding which model to reach for in which situation. Here is the practical landscape.

GPT-4o is OpenAI's flagship. Natively multimodal (text, images, audio in one API call), fast, and strong on most benchmarks. Your default choice when you need maximum capability from the OpenAI ecosystem.

Claude 3.5 and 3.7 Sonnet from Anthropic are the best balance of speed, intelligence, and cost in the Claude family. Particularly strong at following complex, nuanced instructions without going off-script — popular for production systems where reliability matters. Claude 3.7 Sonnet adds "extended thinking" for complex reasoning at the cost of higher latency.

Gemini 2.0 Flash and Pro from Google are notable for native multimodality from the ground up and extremely competitive pricing. Flash is among the most cost-effective capable models available — a default choice for high-volume applications.

Llama 3.x from Meta is the most important open-weight family. The 8B variant runs on consumer hardware. The 70B variant is genuinely competitive with frontier closed models on many tasks. All variants are Apache 2.0 licensed for commercial use.

Mistral and Mixtral push the envelope on open-weight efficiency. Common choice for self-hosted deployments, especially in European enterprises with data sovereignty requirements.

DeepSeek-R1 is an open-weight reasoning model competitive with OpenAI's o1 on math and coding benchmarks. The fact that this capability is now available as downloadable weights is a significant shift.

What This Means for Practitioners: Model Comparison

Practical default choices for this course: Claude Sonnet or GPT-4o for complex tasks. Gemini Flash or GPT-4o-mini for cost-sensitive pipelines. Llama 3 8B/70B for self-hosted experiments. This covers 95% of real-world use cases.

03. Small Language Models (SLMs)

The excitement around GPT-4-scale models can create a false impression that bigger is always better. In many production scenarios, the opposite is true. A small language model — typically under 10 billion parameters — can run on a single consumer laptop, respond in near-real-time, operate entirely offline, and cost essentially nothing per query once you own the hardware.

Microsoft's Phi series demonstrates what carefully chosen training data can accomplish. Phi-3-mini at 3.8B parameters fits in 4GB of VRAM, has a 128K-token context window, and matches or beats Llama 3 8B on many reasoning benchmarks. Phi-4 at 14B scores higher than Llama 3 70B on several math benchmarks while being five times smaller.

Gemma 2 from Google at 2B, 9B, and 27B parameters is specifically optimized for on-device inference. The 2B variant can run on a Raspberry Pi or a modern smartphone.

Quantization makes small models even smaller. Representing weights in 4-bit precision instead of 16-bit reduces memory by 4x. A 7B model goes from 14GB (needs a $10K GPU) to ~3.5GB (runs on a laptop). Quality penalty is typically 1-3% for int8, 3-7% for int4 — often acceptable for production tasks. The dominant format is GGUF, used by llama.cpp and Ollama.

What This Means for Practitioners: When to Choose an SLM

The unit economics are compelling. A frontier API at $3/1M tokens processing 1M daily queries at 500 tokens average costs ~$547K/year. A 7B model on a single GPU at $2.50/hour costs ~$22K/year — a 25x reduction. The tradeoff is quality (which you must measure on your specific task) and operational complexity.

Ollama is the fastest way to run local models. A single command — ollama run llama3.2 — downloads, quantizes, and serves a model. It also exposes an OpenAI-compatible API on localhost, meaning any code written for the OpenAI SDK works unchanged against a local model.

from openai import OpenAI

# Redirect to local Ollama instance
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but ignored by Ollama
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Explain quantization in two sentences."}
    ],
    temperature=0.7,
)
print(response.choices[0].message.content)

04. Multimodal Models

A multimodal model processes multiple types of data — most commonly text and images, but increasingly audio and video. This is not a minor feature addition; it eliminates entire categories of preprocessing pipelines.

The most immediately practical capability is vision: send an image alongside a text prompt and have the model reason about both. Screenshot of an error message? Ask the model to debug it. Photo of a whiteboard? Ask for an explanation. PDF page with tables and charts? Send the page as an image and extract structured data directly — no brittle OCR pipeline needed.

Document understanding is where multimodal models change the game. Traditional PDF extraction meant: convert to text (lossy), run OCR on scans (error-prone), then process with NLP. With multimodal models, you send a page screenshot directly — the model reads text, understands layout, interprets figures, and reasons about everything together.

Audio capabilities add another dimension. OpenAI's Whisper handles speech-to-text with near-human accuracy. GPT-4o's real-time audio API processes speech directly — preserving tone and emotion rather than just transcribing words.

Multimodal input processing: images are split into 16x16 patches, encoded by a Vision Transformer, and projected into the LLM's embedding space. Text is tokenized and embedded separately. Both are concatenated into a single unified sequence before entering the language model decoder.

What This Means for Practitioners: Multimodal Use Cases

Sending images via API requires encoding as base64 and including in the message content as a typed content block:

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

image_data = base64.standard_b64encode(Path("chart.png").read_bytes()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": image_data}},
            {"type": "text", "text": "Extract all data from this chart as JSON."}
        ],
    }],
)
print(message.content[0].text)

Image tokens cost money. A 512x512 image consumes approximately 170 tokens in OpenAI's pricing model. Factor this into your cost estimates for vision-heavy pipelines.

05. Model Selection Framework

One of the most common mistakes is optimizing for the "best" model on general benchmarks. Benchmark performance may have no correlation with performance on your specific task. A model that scores 5 points lower on MMLU might outperform the leader on your use case because it was fine-tuned on similar data or follows your prompt patterns better.

The "good enough" principle is one of the most commercially important ideas in applied AI. A model that achieves 90% of the quality at 20% of the cost and 300% of the speed is almost always the right production choice — provided you can measure that quality gap.

What This Means for Practitioners: The Decision Framework

Start with the latency requirement:

Then check these constraints:

• Vision needed? GPT-4o, Claude 3+, Gemini 1.5+, or Llama 3.2 (open-weight)
• Private data? Self-host: Llama 3, Mistral, Phi-4, Qwen via Ollama or vLLM
• Multilingual? Qwen 2.5 (strong CJK), Claude (strong European), GPT-4o (broad)
• Budget-constrained high volume? Gemini Flash ($0.075/1M in) or self-hosted SLM

The professional testing methodology:

1. Define your task and collect 50-200 representative examples
2. Label them with correct outputs (human-annotated or from a trusted model)
3. Run each candidate model with your production prompt
4. Score outputs against golden labels
5. Sort by quality-per-dollar on your specific task
6. Pick the model at the knee of the curve

import tiktoken

def estimate_cost(prompt: str, model: str = "gpt-4o") -> dict:
    PRICING = {
        "gpt-4o": (2.50, 10.00), "gpt-4o-mini": (0.15, 0.60),
        "claude-sonnet-4-6": (3.00, 15.00), "gemini-2.0-flash": (0.075, 0.30),
    }
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        encoding = tiktoken.get_encoding("cl100k_base")
    input_tokens = len(encoding.encode(prompt))
    price_in, price_out = PRICING.get(model, (3.0, 15.0))
    estimated_output = 300
    return {
        "input_tokens": input_tokens,
        "est_cost_usd": round((input_tokens / 1_000_000) * price_in + (estimated_output / 1_000_000) * price_out, 6),
    }

Interview Ready

Interview Questions

Model Answers

Common Mistakes

• Defaulting to the largest model for every task. GPT-4o for simple classification wastes money and adds latency. Measure quality on your specific task — GPT-4o-mini or a fine-tuned 7B model is often sufficient.
• Ignoring parameter count when planning infrastructure. A 70B model at FP16 requires ~140GB VRAM — two A100 GPUs minimum. Proposals to "just deploy Llama 70B" without hardware planning reveal a gap between theory and production readiness.
• Treating multimodal as OCR + LLM. Native multimodal models process layout, figures, and text together. A pipeline of "OCR the document, then send text to an LLM" misses the architectural shift — direct image input handles tables and charts far more reliably.

Previous Module

01 · Foundations of GenAI

Next Module

03 · APIs for LLMs

Phase: Foundations → API Access