Part IV — Enterprise AI Architecture

Observability — Seeing Inside the Black Box

Traditional application observability answers: is the system up? Is it slow? Is it returning errors? These questions are necessary but insufficient for AI systems. An AI system can be fully operational — returning responses with 200ms latency and zero HTTP errors — while silently producing incorrect, biased, or harmful outputs at scale. AI observability adds a layer that answers: is the system producing correct outputs? Is the quality degrading? Are the inputs changing in ways that will cause future quality problems?

43.1 The Three Layers of AI Observability

Layer 1 — Infrastructure observability. The standard metrics of any distributed system: API availability, latency percentiles (p50, p95, p99), error rates, token consumption, cost per inference, request volume. Infrastructure observability is table stakes — these metrics should be instrumented from day one and integrated into existing monitoring infrastructure (Datadog, Grafana, CloudWatch).

Layer 2 — Quality observability. Measurement of output quality in production — whether the system is producing outputs that are correct, relevant, and appropriate. Quality observability requires:

Sampling: collect a sample of production inputs and outputs (typically 1–5%) for quality evaluation.

Automated scoring: run automated quality checks on sampled outputs — heuristic checks (format compliance, required field presence), LLM-as-judge scoring (semantic quality, relevance, accuracy), and embedding similarity against reference outputs.

Trend tracking: track quality scores over time and alert when scores deviate significantly from baseline. A quality score that is declining over time is a leading indicator of an emerging problem.

Layer 3 — Input distribution observability. Monitoring how the distribution of production inputs changes over time. Input distribution shift — users querying about topics not represented in the evaluation dataset, document types the model was not trained on, languages outside the model's primary training language — is a leading indicator of quality degradation that becomes visible in quality observability only after the degradation is significant.

Input distribution monitoring uses statistical tests (Kolmogorov-Smirnov test, Population Stability Index) to detect when the current input distribution differs significantly from the baseline. This is an early warning system, not a quality measurement.

43.2 Tracing Multi-Step AI Systems

Modern AI architectures are multi-component systems: retrieval pipelines, LLM calls, tool invocations, output parsers, evaluation steps. When something goes wrong, attributing the failure to the correct component requires distributed tracing across all components.

LLM-specific tracing. Standard distributed tracing tools (Jaeger, Zipkin, OpenTelemetry) must be extended for LLM calls with AI-specific attributes: the model name and version, the prompt template identifier, the token counts, the cost, and a hash or excerpt of the input and output for debugging.

RAG pipeline tracing. For RAG systems, trace the complete retrieval pipeline: what query was embedded, what chunks were retrieved, what metadata filters were applied, what context was assembled. When a response is wrong, RAG tracing determines whether the failure was in retrieval (wrong chunks) or generation (the right chunks were retrieved but the LLM ignored them or reasoned incorrectly about them).

Agentic system tracing. For agentic systems, trace each reasoning step: what observation triggered the current step, what the agent's plan was, what tool was selected, what the tool returned, and how the agent updated its plan. Agentic traces are often long and branching; tooling that supports visualization of branching execution paths is essential for debugging.

Correlation IDs. Every inference request should carry a correlation ID that flows through all downstream component calls. Correlation IDs allow an operator to reconstruct the complete execution path for any specific request — essential for investigating user-reported issues.

43.3 Quality Degradation Signals

Quality degradation in AI systems tends to present through specific observable signals before it becomes catastrophically user-visible:

Increasing edit rates. If users who accept AI-generated drafts increasingly edit them before use, the edits are a quality signal. Measuring the magnitude of edits (character-level or semantic distance between the AI draft and the final submitted version) provides a continuous quality signal from user behavior.

Declining acceptance rates. If users are presented with AI suggestions and an explicit accept/reject action, declining acceptance rates indicate degrading quality before the decline appears in quality evaluation scores.

Rising user corrections. In systems where users correct AI errors (selecting the wrong categorization, overriding an AI decision), correction rates are a quality signal. Corrections should be logged with both the AI's output and the user's correction for future evaluation dataset use.

Increasing latency on specific query types. Some AI systems show latency degradation on specific query patterns as they scale — particularly agentic systems where certain query types trigger longer tool-use chains. Latency segmented by query type surfaces these patterns.

Token consumption growth. If average token consumption per query is increasing over time without a corresponding increase in task complexity, it may indicate prompt drift (the prompt has been expanded without removing old content), context retrieval tuning issues, or output constraint violations.

43.4 Dashboards and Alerting

AI observability without dashboards and alerting is data collection, not observability. Operational teams need to see the signal without drowning in noise.

The AI operations dashboard:

• Infrastructure health: availability, latency p50/p95/p99, error rate, cost per day
• Quality score trend: rolling 7-day and 30-day quality score vs. baseline
• Input distribution drift score: statistical distance from baseline distribution
• User feedback signals: acceptance rate, correction rate, edit magnitude
• Model version and prompt version currently in production

Alerting thresholds:

Alert on: quality score drops more than 10% from the 30-day baseline; input distribution drift score exceeds the defined threshold; cost per inference increases more than 25% from the 7-day baseline; latency p99 exceeds the SLA target; error rate exceeds 1% on a sustained basis.

Alert fatigue management. AI observability alerting that fires too frequently produces alert fatigue — operators learn to ignore alerts because most turn out to be false positives. Define alert thresholds conservatively, tune them against historical data, and require corroboration (two independent signals) before paging an operator.

AI observability is what converts AI systems from black boxes into maintainable production infrastructure. Teams that build observability from the start have the visibility to diagnose problems quickly; teams that skip it spend weeks debugging failures that would have taken hours to identify with proper instrumentation.