AI Observability: Debugging Systems That Do Not Fail Consistently

Debugging conventional software often begins with a recognizable signal. A database connection might time out, an API might return an error code, or a function might receive the wrong data type. Engineers can inspect logs and reproduce the conditions that caused the failure.

AI systems are often harder to diagnose. The same request may succeed several times and then fail without an obvious technical error. A model might choose the wrong tool, retrieve an irrelevant document, generate malformed arguments, or produce a confident answer that its sources do not support. Every service can appear healthy while the final result is still wrong.

AI observability helps engineering teams reconstruct what happened. It leaves a trail of breadcrumbs connecting familiar infrastructure telemetry with the prompts, retrieval decisions, tool calls, evaluations, and policy checks that shaped the response.

Why AI Failures Are Difficult to Reproduce

If you’ve kept up with Onyx’s previous posts, you know that language model outputs are probabilistic. Sampling can produce different responses to the same prompt, and model updates may change behavior without any changes to the application itself. The surrounding context also evolves as conversations grow, documents are updated, and tools return new information.

Agentic workflows introduce even more variation. One run might call a search tool, while another queries a database. A failed tool call can influence every decision that follows. Looking only at the final response leaves engineers guessing about where the problem began.

Tracing the Complete AI Workflow

A trace groups the recorded operations associated with a request or workflow. With proper instrumentation and context propagation, it can follow that request across multiple services. Individual spans capture specific operations, including model generations, retrieval queries, tool calls, guardrail checks, and evaluator runs.

Useful telemetry for a model call may include:

Retrieval spans should record details such as the query, index version, filters, document identifiers, relevance scores, and passages added to the model's context. Tool spans should capture validated arguments, execution status, latency, and a safely filtered result.

Emerging standards are beginning to formalize these fields. OpenTelemetry’s generative AI semantic conventions define common attributes for model operations, usage, and responses. Frameworks can also connect model generations with tool calls and guardrails, as shown by the OpenAI Agents SDK tracing model.

The relationships between events matter as much as the events themselves. Trace IDs, span IDs, and propagated context help engineers connect model responses, tool requests, external results, and final answers across instrumented services.

Operational Health Does Not Measure Answer Quality

Latency, error rates, and resource utilization remain valuable, but they do not reveal whether an answer is accurate or grounded. An AI application can return successful HTTP responses all day while quietly producing poor results.

Quality signals need to become part of the production trace. Depending on the application, these signals might measure citation support, retrieval relevance, policy compliance, tool-selection accuracy, structured-output validity, or task completion. Some can be calculated automatically. Others require feedback from users or subject matter experts.

Each signal measures a limited property. Schema validation makes sure an output has the correct structure, but it says little about the accuracy of the content. Automated relevance and citation scores also depend on the quality of the evaluator. Evaluations become more useful when teams look for patterns across many traces. A model might perform well overall while struggling with long documents, uncommon acronyms, or a particular tool. Segmenting results by task, model version, prompt version, and data source helps expose these clusters.

Turning Production Failures Into Evaluation Cases

Exact reproduction may be impossible after a model, retrieval index, or external data source changes. Teams can improve repeatability by versioning prompts, tool schemas, policy rules, and retrieval indexes. They should also pin model versions where supported and record available configuration details for external dependencies.

Captured inputs and relevant state can then become evaluation cases for future configurations. Preserved documents or recorded tool responses may be needed to control changing dependencies. Because model behavior can vary between runs, repeated trials may provide a better picture of whether a change reduces intermittent failures or creates new ones.

This process turns isolated incidents into a durable test suite built from problems that users actually experienced.

Observability Creates a Data Security Problem

Detailed traces may contain prompts, retrieved documents, personal information, credentials, and internal tool outputs. Collecting all of that data can create a sensitive copy of the production environment.

Telemetry pipelines should redact secrets, restrict access, encrypt stored traces, and enforce retention limits. When raw content is unnecessary, teams may store metadata or hashes. Sampling policies can preserve detailed traces for detected failures and sensitive workflows, retain a representative sample of successful requests, and collect aggregate metrics for the remaining traffic.

The NIST AI Risk Management Framework Playbook emphasizes measurement and monitoring throughout the AI lifecycle. The data collected for monitoring still needs to follow applicable security, privacy, access, and retention requirements.

Production Takeaways

AI observability works best when both system behavior and output quality are captured:

Inconsistent failures cannot be understood from final responses alone. Engineers need to see what the model received, which actions it requested, what external systems returned, and how the result was evaluated. A well-designed observability layer makes those steps visible and gives teams a practical way to turn unpredictable behavior into measurable improvements.