AI Observability & Evals
This domain accounts for 20% of the AI-ADV exam and tests your ability to instrument, monitor, evaluate, and continuously improve AI applications in production. Questions focus on tool selection (Langfuse vs. Braintrust vs. Arize vs. Phoenix), architectural decisions for tracing and logging, evaluation pipeline design, cost tracking, and human feedback integration. Expect scenario-based questions requiring you to choose the right tool or approach given a specific constraint.
Why Observability Matters for AI
Traditional software observability assumes deterministic behavior: the same input always produces the same output. AI applications break this assumption in three critical ways.
Non-determinism means an LLM can produce different outputs for identical inputs due to temperature settings, sampling strategies, and model updates. A bug may appear only 5% of the time, making it invisible to conventional pass/fail tests.
Prompt drift occurs when a model update, context window change, or upstream data shift causes prompt performance to degrade silently over time. A prompt that scored 0.92 on quality in January may score 0.74 in March after a model provider silently updates their weights.
Silent failures are the most dangerous: the system returns a 200 OK with a plausible-sounding but factually wrong or off-policy response. No exception is thrown. Without observability, these failures reach users undetected.
| Failure Type | Detectable Without Observability? | Primary Detection Method |
|---|---|---|
| Non-determinism variance | No | Distribution tracking over many calls |
| Prompt drift | No | Regression eval on golden dataset |
| Silent hallucination | No | LLM-as-judge or human scoring |
| Token limit exceeded | Yes (exception) | Error logging |
| Latency spike | Partially (timeout) | Span-level latency tracing |
Common traps:
- Assuming unit tests catch silent failures — they do not; unit tests only catch exceptions and schema errors.
- Confusing prompt drift with model versioning; drift can occur even when you pin a model version if the provider updates weights under the same version label.
- Treating high accuracy on a holdout set as sufficient for production — production distribution shift is not captured by static eval sets.
OpenTelemetry for AI
OpenTelemetry (OTel) is the CNCF standard for distributed tracing, metrics, and logs. For AI applications, it provides vendor-neutral instrumentation for LLM calls.
Core Concepts
A trace represents a complete request through the system (e.g., user query → retrieval → LLM call → response). A trace contains one or more spans. Each span represents a single unit of work with a start time, end time, and attributes.
For AI workloads, the key OTel semantic conventions (under gen_ai.* namespace) include:
| Attribute | Description | Example Value |
|---|---|---|
gen_ai.system | LLM provider | openai, anthropic, google |
gen_ai.request.model | Model requested | gpt-4o |
gen_ai.response.model | Model actually used | gpt-4o-2024-08-06 |
gen_ai.usage.input_tokens | Prompt token count | 512 |
gen_ai.usage.output_tokens | Completion token count | 128 |
gen_ai.request.temperature | Sampling temperature | 0.7 |
gen_ai.response.finish_reason | Why generation stopped | stop, length, content_filter |
Spans in an AI pipeline are typically nested: a root span for the HTTP request, child spans for retrieval, and a grandchild span (a generation span) for the LLM call itself.
Common traps:
- Confusing
gen_ai.request.modelandgen_ai.response.model— they differ when a provider routes to a different underlying model. - Assuming OTel alone provides evaluation scores — OTel captures telemetry; scoring is a separate layer (Langfuse, Braintrust).
- Forgetting that OTel is the collection standard; you still need a backend (Jaeger, Langfuse, Honeycomb) to store and query traces.
Worked scenario: A team uses three different LLM providers and wants vendor-neutral tracing they can route to any backend. → Use OTel with gen_ai.* semantic conventions; the exporter can be swapped without changing instrumentation code.
Unlock with a paid plan
12 more sections (about 2,662 words) continue below. Every paid plan unlocks all AI Advanced study guides, the full question bank and mock exams, from $4.97/month.