Part 4: Advanced Patterns

Evaluating and Validating LLM Outputs

An LLM that sounds authoritative is not the same as an LLM that is correct. Every analyst who has used ChatGPT has experienced the unsettling moment when a perfectly fluent, well-structured response turns out to be confidently wrong. In production workflows where LLM outputs feed into business decisions, compliance documents, or test plans, undetected errors carry real consequences. This chapter gives you a systematic framework for evaluating, validating, and building trust in LLM-generated outputs — from automated metrics to human-in-the-loop review processes.

Reading time: ~25 min Project: Validation Framework

Figure 15.1 — The three-layer validation framework. Automated checks handle high volume at the bottom. LLM-as-Judge catches semantic issues in the middle. Human review handles edge cases and critical decisions at the top.

Figure 15.2 — A/B testing for LLM workflows. The same inputs are processed by two prompt versions, outputs are scored, and statistical comparison identifies the winner.

15.1 The Validation Imperative

LLMs are probabilistic text generators. They do not "know" anything. They predict the next token based on statistical patterns in their training data. Every output is a prediction, not a fact. For business analysts producing requirements documents or quality analysts generating test cases, this distinction matters enormously.

Consider the risk profile of different LLM use cases:

The validation level should match the risk level. Using an LLM to brainstorm feature ideas requires no validation — a bad idea is easily discarded. Using an LLM to generate regulatory compliance documentation requires multiple validation layers because a wrong claim could result in legal liability.

The core validation question is always the same: How do you know this output is correct? If you cannot answer that question for your use case, you are not ready to put the LLM output into production.

The most dangerous property of LLMs is that wrong answers are just as fluent and well-structured as right answers. A hallucinated requirement reads exactly like a real requirement. A fabricated test case looks exactly like a valid test case. You cannot rely on "it sounds right" as a validation strategy. You need systematic, repeatable validation processes.

15.2 Accuracy Metrics for Text

Measuring the accuracy of generated text is harder than measuring classification model accuracy (where you count correct predictions). Text accuracy has multiple dimensions: factual correctness, completeness, format compliance, and semantic equivalence.

Automated validation starts with deterministic checks that require no LLM calls: verify the output is valid JSON (if expected), check that text length falls within acceptable bounds, confirm required fields are present, and validate against a schema. These checks are fast, free, and catch the most obvious failures. A format validator can check dozens of outputs per second, making it practical to validate every single LLM response in production. When a check fails, the system can automatically retry with a clarified prompt before escalating to LLM-based or human review.

Key accuracy dimensions and how to measure each:

Create a set of 50–100 "golden" input-output pairs where you know the correct answer. Run every model change, prompt change, or pipeline change against the golden dataset and compare scores. This is your regression test suite for LLM output quality.

15.3 Hallucination Detection

Hallucination is the LLM failure mode that matters most for enterprise use cases. A hallucination occurs when the LLM generates information that is not grounded in its input (for RAG systems) or that is factually incorrect (for general generation). Automated hallucination detection is one of the most valuable validation capabilities you can build.

from openai import OpenAI
import json
import re

client = OpenAI()

class HallucinationDetector:
    """Detect various types of hallucination in LLM outputs."""

    def __init__(self, model: str = "gpt-4o"):
        self.model = model

    def detect_ungrounded_claims(self, output: str,
                                   source_documents: list[str]) -> dict:
        """Find claims in the output that are not supported
        by the source documents (for RAG systems)."""
        sources = "\n---\n".join(source_documents)

        response = client.chat.completions.create(
            model=self.model,
            messages=[{
                "role": "user",
                "content": f"""Analyze the Output for hallucinations.
A hallucination is any claim in the Output that is NOT supported
by the Source Documents.

Source Documents:
{sources}

Output to check:
{output}

For each claim in the output, determine:
1. claim: The specific claim made
2. grounded: Is it directly supported by the sources? (yes/no)
3. evidence: Quote the supporting text from sources, or "none"
4. severity: If ungrounded, how risky is this claim?
   (low = stylistic, medium = potentially misleading,
    high = factually wrong and could cause harm)

Return JSON with:
- "claims": array of claim objects
- "grounded_count": number of grounded claims
- "hallucinated_count": number of ungrounded claims
- "hallucination_rate": 0.0-1.0
- "high_severity_hallucinations": array of dangerous claims"""
            }],
            response_format={"type": "json_object"},
            temperature=0
        )
        return json.loads(response.choices[0].message.content)

    def detect_fabricated_references(self, output: str) -> dict:
        """Detect fabricated citations, URLs, document names,
        or statistics that the LLM may have invented."""
        response = client.chat.completions.create(
            model=self.model,
            messages=[{
                "role": "user",
                "content": f"""Analyze this text for potentially
fabricated references. Look for:
1. Citations to papers, books, or documents (are they real?)
2. URLs (do they look real or invented?)
3. Statistics and numbers (are they specific enough to be
   verifiable, and do they seem plausible?)
4. Named individuals or organizations in specific claims
5. Dates of events or decisions

For each reference found:
- reference: The reference text
- type: citation/url/statistic/person/date
- suspicion_level: low/medium/high
- reason: Why it might be fabricated

Text:
{output}

Return JSON with key "references" (array) and
"fabrication_risk_score" (0.0-1.0)."""
            }],
            response_format={"type": "json_object"},
            temperature=0
        )
        return json.loads(response.choices[0].message.content)

    def detect_internal_contradictions(self, output: str) -> dict:
        """Find statements within the output that contradict
        each other."""
        response = client.chat.completions.create(
            model=self.model,
            messages=[{
                "role": "user",
                "content": f"""Analyze this text for internal
contradictions — places where the text says two things that
cannot both be true.

Examples of contradictions:
- "The system supports 100 users" ... later ...
  "Maximum capacity is 50 concurrent users"
- "This feature is mandatory" ... later ...
  "This feature is optional for Phase 1"

Text:
{output}

Return JSON with:
- "contradictions": array of {{statement_1, statement_2,
  explanation}}
- "contradiction_count": integer
- "has_contradictions": boolean"""
            }],
            response_format={"type": "json_object"},
            temperature=0
        )
        return json.loads(response.choices[0].message.content)

    def comprehensive_check(self, output: str,
                              source_documents: list[str] = None) -> dict:
        """Run all hallucination checks and produce a report."""
        results = {
            "fabricated_references": self.detect_fabricated_references(
                output
            ),
            "internal_contradictions":
                self.detect_internal_contradictions(output),
        }

        if source_documents:
            results["ungrounded_claims"] = self.detect_ungrounded_claims(
                output, source_documents
            )

        # Overall risk assessment
        risk_score = 0.0
        if source_documents:
            risk_score += results["ungrounded_claims"].get(
                "hallucination_rate", 0
            ) * 0.5
        risk_score += results["fabricated_references"].get(
            "fabrication_risk_score", 0
        ) * 0.3
        risk_score += (
            0.2 if results["internal_contradictions"].get(
                "has_contradictions", False
            ) else 0
        )

        results["overall_risk_score"] = round(risk_score, 3)
        results["recommendation"] = (
            "SAFE" if risk_score < 0.2 else
            "REVIEW" if risk_score < 0.5 else
            "REJECT"
        )

        return results


# Usage
detector = HallucinationDetector()

# Check a RAG-generated answer
answer = """According to our data retention policy (DRP-2024-v3),
customer PII must be deleted within 90 days of account closure.
The GDPR requires this under Article 17 (Right to Erasure).
Our compliance team confirmed in Q3 2025 that we are 98.5%
compliant across all EU data centers."""

sources = [
    "Data Retention Policy DRP-2024-v3: Customer personally "
    "identifiable information (PII) shall be purged within "
    "30 days of account closure or upon written request.",
    "GDPR Article 17 establishes the right to erasure, requiring "
    "data controllers to delete personal data without undue delay."
]

results = detector.comprehensive_check(answer, sources)

print(f"Overall risk: {results['overall_risk_score']}")
print(f"Recommendation: {results['recommendation']}")
if results.get("ungrounded_claims"):
    rate = results["ungrounded_claims"]["hallucination_rate"]
    print(f"Hallucination rate: {rate:.1%}")
    for claim in results["ungrounded_claims"].get(
        "high_severity_hallucinations", []
    ):
        print(f"  HIGH RISK: {claim}")

In the example above, the detector should flag two issues: the policy says 30 days but the answer says 90 days (factual error), and the "98.5% compliant" statistic is not in the source documents (fabricated statistic). Both are dangerous hallucinations that could lead to compliance violations.

LLMs hallucinate more when questions are ambiguous, when the topic is at the boundary of the training data, when the prompt asks for specific numbers or dates, and when the context window is near capacity. Track which types of queries produce the most hallucinations and add extra validation for those categories.

15.4 Consistency Checking

LLMs are non-deterministic by design. Ask the same question twice and you may get different answers. BA and QA workflows often require consistency: the same requirement should always produce the same test case format, and the same defect should always receive the same severity rating. Consistency checks enforce this.

Consistency checking runs the same input through the LLM multiple times (typically 3–5 runs) and compares the outputs. For classification tasks, you check whether the label is the same across runs. A requirement classified as "high priority" in 3 of 5 runs but "medium" in 2 runs signals low confidence. For generation tasks, you use an LLM-as-judge to score semantic similarity between runs. A consistency score below 0.7 flags the output for human review. This catches cases where the LLM is uncertain, even when each individual output looks confident.

Strategies for improving consistency:

Set temperature to 0 for classification tasks. When the LLM is classifying defect severity, categorizing requirements, or making yes/no decisions, you want the same input to always produce the same classification. Save higher temperatures (0.5–0.8) for generative tasks where variety is desirable, like brainstorming test scenarios.

15.5 Human-in-the-Loop Validation

Automated validation catches many errors, but some require human judgment. Is this requirement actually what the stakeholder meant? Does this test case cover the real risk? Is this defect description clear to a developer? Human-in-the-loop (HITL) validation combines the scale of automation with the judgment of domain experts.

The human-in-the-loop review system routes flagged outputs to domain experts through a review queue. Each review item includes the original input, LLM output, confidence score, and the specific validation check that triggered the review. Reviewers can approve, reject, or edit outputs. Rejected items include a reason category (factual error, incomplete, wrong format, inappropriate tone) that feeds back into prompt improvement. Track reviewer agreement rates — if two reviewers disagree frequently on the same category, your validation criteria need tightening.

A well-designed HITL process balances thoroughness with efficiency:

Randomly sample 10% of auto-approved items each week and manually review them. If more than 5% would have been modified or rejected by a human, your auto-approve threshold is too low. This feedback loop ensures your automation does not silently degrade quality.

15.6 A/B Testing LLM Workflows

When you change a prompt, switch models, or modify the pipeline, how do you know the change is an improvement? A/B testing provides statistical evidence. You run both versions on the same inputs and measure which performs better.

import random
import json
from datetime import datetime
from dataclasses import dataclass, field
from openai import OpenAI

client = OpenAI()

@dataclass
class ABTestConfig:
    """Configuration for an A/B test of LLM workflows."""
    test_name: str
    variant_a: dict  # {"name": "...", "model": "...", "prompt": "..."}
    variant_b: dict
    metrics: list[str]  # ["accuracy", "latency", "cost", "preference"]
    sample_size: int = 100
    traffic_split: float = 0.5  # 50/50 by default

@dataclass
class ABTestResult:
    """Result of a single test case in an A/B test."""
    test_case_id: str
    variant: str  # "A" or "B"
    input_text: str
    output: str
    metrics: dict
    timestamp: str = field(
        default_factory=lambda: datetime.now().isoformat()
    )

class ABTestRunner:
    """Run A/B tests on LLM workflow variants."""

    def __init__(self):
        self.results: list[ABTestResult] = []

    def run_test(self, config: ABTestConfig,
                 test_cases: list[dict],
                 evaluator: TextAccuracyEvaluator) -> dict:
        """Run an A/B test across all test cases."""
        self.results = []

        for i, tc in enumerate(test_cases):
            # Assign variant
            variant = ("A" if random.random() < config.traffic_split
                       else "B")
            variant_config = (config.variant_a if variant == "A"
                              else config.variant_b)

            # Run the variant
            import time
            start = time.time()

            response = client.chat.completions.create(
                model=variant_config["model"],
                messages=[
                    {"role": "system",
                     "content": variant_config["prompt"]},
                    {"role": "user", "content": tc["input"]}
                ],
                temperature=variant_config.get("temperature", 0.3)
            )

            output = response.choices[0].message.content
            latency = time.time() - start
            tokens = response.usage.total_tokens

            # Evaluate
            metrics = {
                "latency_seconds": round(latency, 3),
                "tokens": tokens,
                "cost_usd": round(tokens * 0.000005, 6),
                "output_length": len(output),
            }

            # If reference answer exists, measure accuracy
            if tc.get("reference"):
                accuracy = evaluator.evaluate_factual_accuracy(
                    output, tc["reference"]
                )
                metrics["accuracy_score"] = accuracy.get(
                    "accuracy_score", 0
                )
                metrics["completeness_score"] = accuracy.get(
                    "completeness_score", 0
                )

            self.results.append(ABTestResult(
                test_case_id=f"TC-{i+1}",
                variant=variant,
                input_text=tc["input"][:100],
                output=output,
                metrics=metrics
            ))

            if (i + 1) % 10 == 0:
                print(f"  Completed {i+1}/{len(test_cases)} test cases")

        return self._analyze_results(config)

    def _analyze_results(self, config: ABTestConfig) -> dict:
        """Statistical analysis of A/B test results."""
        a_results = [r for r in self.results if r.variant == "A"]
        b_results = [r for r in self.results if r.variant == "B"]

        def avg_metric(results, metric):
            values = [r.metrics.get(metric, 0) for r in results]
            return round(sum(values) / len(values), 4) if values else 0

        analysis = {
            "test_name": config.test_name,
            "total_samples": len(self.results),
            "variant_a": {
                "name": config.variant_a["name"],
                "count": len(a_results),
            },
            "variant_b": {
                "name": config.variant_b["name"],
                "count": len(b_results),
            },
            "comparisons": {}
        }

        # Compare each metric
        for metric in ["accuracy_score", "completeness_score",
                       "latency_seconds", "tokens", "cost_usd"]:
            a_avg = avg_metric(a_results, metric)
            b_avg = avg_metric(b_results, metric)
            diff_pct = (
                round((b_avg - a_avg) / a_avg * 100, 1)
                if a_avg > 0 else 0
            )

            # For latency and cost, lower is better
            lower_is_better = metric in [
                "latency_seconds", "tokens", "cost_usd"
            ]
            winner = (
                "B" if (b_avg < a_avg) == lower_is_better else "A"
            )

            analysis["comparisons"][metric] = {
                "variant_a": a_avg,
                "variant_b": b_avg,
                "difference_pct": diff_pct,
                "winner": winner
            }

        # Overall recommendation
        a_wins = sum(
            1 for c in analysis["comparisons"].values()
            if c["winner"] == "A"
        )
        b_wins = sum(
            1 for c in analysis["comparisons"].values()
            if c["winner"] == "B"
        )
        analysis["recommendation"] = (
            f"Variant {'A' if a_wins > b_wins else 'B'} "
            f"({config.variant_a['name'] if a_wins > b_wins else config.variant_b['name']}) "
            f"wins on {max(a_wins, b_wins)}/{a_wins + b_wins} metrics"
        )

        return analysis


# Example: Compare two prompt variants for user story generation
ab_config = ABTestConfig(
    test_name="User Story Prompt Comparison",
    variant_a={
        "name": "Concise Prompt",
        "model": "gpt-4o",
        "prompt": "Generate a user story from this requirement. "
                  "Use format: As a [role], I want [what], "
                  "so that [why]. Include acceptance criteria.",
        "temperature": 0.3
    },
    variant_b={
        "name": "Detailed Prompt with Examples",
        "model": "gpt-4o",
        "prompt": """Generate a user story from this requirement.

Format:
Title: [concise title]
Story: As a [specific role], I want [specific capability],
so that [measurable benefit].
Acceptance Criteria:
- Given [context], When [action], Then [outcome]
(include 3-5 criteria covering happy path and edge cases)
Story Points: [1/2/3/5/8/13]

Example:
Title: Password Reset via Email
Story: As a registered customer, I want to reset my password
via email, so that I can regain account access within 5 minutes.
Acceptance Criteria:
- Given I am on the login page, When I click "Forgot Password"
  and enter my email, Then I receive a reset link within 60 seconds
- Given I have a reset link, When I click it after 24 hours,
  Then I see an "expired link" message""",
        "temperature": 0.3
    },
    metrics=["accuracy", "completeness", "latency", "cost"],
    sample_size=50
)

test_cases = [
    {"input": "Users need to be able to export reports to PDF",
     "reference": "As a report viewer, I want to export reports "
                  "to PDF format, so that I can share them offline."},
    {"input": "The system should send email notifications when "
              "an order ships",
     "reference": "As a customer, I want to receive an email "
                  "when my order ships, so that I can track delivery."},
    # Add more test cases...
]

runner = ABTestRunner()
evaluator = TextAccuracyEvaluator()
results = runner.run_test(ab_config, test_cases, evaluator)

print(f"\n{results['test_name']}")
print("=" * 50)
for metric, comparison in results["comparisons"].items():
    print(f"  {metric}:")
    print(f"    A ({results['variant_a']['name']}): "
          f"{comparison['variant_a']}")
    print(f"    B ({results['variant_b']['name']}): "
          f"{comparison['variant_b']}")
    print(f"    Winner: Variant {comparison['winner']} "
          f"({comparison['difference_pct']:+.1f}%)")
print(f"\nRecommendation: {results['recommendation']}")

With 10 test cases, a 5% accuracy difference could be noise. With 100 test cases, a 5% difference is meaningful. Run at least 50 test cases per variant before drawing conclusions. For high-stakes decisions (switching models, changing production prompts), aim for 200+ test cases and compute confidence intervals.

15.7 Building Trust with Stakeholders

The technical validation framework means nothing if stakeholders do not trust the outputs. Building trust is a communication and change-management challenge as much as a technical one. Stakeholders need to understand what the AI can and cannot do, see evidence of quality, and feel in control.

Building stakeholder trust requires three practices: transparency (always show the confidence score and explain what validation checks were applied), provenance (cite the source documents or data that informed each output), and progressive disclosure (start with low-stakes tasks where errors are cheap, demonstrate reliability, then expand to higher-stakes workflows).

The trust-building journey follows a predictable pattern:

The biggest trust-builder is the ability to be wrong gracefully. When the AI makes a mistake — and it will — how the system handles it determines whether stakeholders lose trust or gain it. An assistant that says "I am not confident about this claim, please verify the retention period in the DRP policy" builds more trust than one that states the wrong number confidently. Design your system to express uncertainty rather than hide it.

Project: Validation Framework

Build a comprehensive validation framework that can be plugged into any LLM workflow to evaluate, validate, and report on output quality. The framework should combine automated metrics, hallucination detection, consistency checks, human review, and stakeholder reporting.

Project Requirements

1. Implement automated accuracy evaluation against a golden dataset of at least 20 input-output pairs
2. Build a hallucination detector that checks for ungrounded claims, fabricated references, and internal contradictions
3. Add consistency checking that runs the same prompt 5 times and measures output stability
4. Implement a review queue with auto-approve, auto-reject, and human review tiers
5. Build an A/B test runner that can compare two prompt or model variants on the golden dataset
6. Generate a stakeholder transparency report summarizing all quality metrics
7. Log all validation results for trend analysis over time

Starter Code

Your Validation Framework project combines all four validation layers: automated format and schema checks (Layer 1), LLM-as-judge scoring for relevance, coherence, and factuality (Layer 2), consistency checking across multiple runs (Layer 2.5), and a human review queue for flagged items (Layer 3). Configure confidence thresholds per use case: a test case generator might accept 0.75 confidence, while a compliance document generator requires 0.95. Run the framework against your Chapter 9 test case outputs or Chapter 5 requirements analysis outputs to see real validation results.

Extension Ideas

• Add a web dashboard (Streamlit) that displays the transparency report with interactive charts showing quality trends over time
• Implement "regression alerts" that notify the team when quality metrics drop below thresholds (e.g., accuracy drops from 95% to 88% after a prompt change)
• Build a feedback loop where rejected items and human corrections are automatically added to the golden dataset
• Add domain-specific validators: for user stories, check that every story follows INVEST criteria; for test cases, check that every case has at least one assertion
• Implement cost-aware validation: skip expensive checks (LLM-as-judge) for low-risk outputs and reserve them for high-stakes content

Exercises

1. Golden dataset. Create a golden dataset of 20 input-output pairs for a workflow you use regularly (user story generation, test case creation, or defect analysis). Run your current LLM pipeline against it and measure baseline accuracy.
2. Hallucination hunt. Generate 10 outputs from your LLM workflow and manually check each one for hallucinations. Categorize each hallucination (fabricated fact, wrong number, invented reference). Then run the automated detector on the same outputs. How many did it catch?
3. Consistency test. Pick a prompt you use frequently. Run it 10 times at temperature 0.3 and 10 times at temperature 0. Measure the consistency score for each. Is the difference significant?
4. Review queue design. Design a review queue for your team's use case. Define the auto-approve threshold, required reviewers per content type, and escalation criteria. Simulate 50 items flowing through the queue and calculate reviewer workload.
5. Transparency report. Generate a transparency report for a workflow you have been using for at least two weeks. Share it with a skeptical stakeholder. What questions do they ask? What additional information would increase their confidence?