AI Advanced: AI Observability & Evals Practice Questions

Confounding prompt version with time is correct. Running each variant in its own time window causes temporal confounding: differences in traffic patterns, user intent, or outside events between the two periods cannot be told apart from the prompt's effect. A proper A/B test randomly assigns requests to the variants at the same time. The claim that two weeks is always too short is wrong in absolute terms: two weeks may be enough depending on traffic volume, because significance depends on sample size, not calendar duration alone. The claim that the model version cannot be guaranteed across periods is a false constraint: model version is a separate variable that can be controlled (for example, by pinning it) regardless of the time period. The claim that sequential testing violates provider terms of service is false: no major LLM provider prohibits A/B testing of prompts.

Propagating trace context is correct: trace correlation in OpenTelemetry comes from context propagation — each child span carries the parent's trace_id and span_id, forming a causal tree. Without propagating this context across service boundaries or async steps, spans appear as disconnected root spans. The synchronized-timestamp answer is wrong — timestamps do not link spans; two unrelated spans can share a timestamp. The same-collector answer is incorrect — the collector is a transport mechanism, and multiple collectors can receive spans from the same trace. The shared-resource-attributes answer is wrong because resource attributes affect service identity in the backend but do not create parent-child relationships between spans.

Logging the prompt or a versioned hash with each call is correct: teams can then segment quality metrics by prompt version and see when a particular version is associated with score degradation, making drift visible before it becomes a major incident. The CPU/GPU/memory monitoring answer is false — prompt drift affects semantic quality, not infrastructure resource consumption. The self-evaluation score answer can add noise, since models tend to rate themselves highly regardless of actual quality; it also doesn't identify the prompt version as the cause. The support-ticket answer relies on a reactive, lagging signal that detects drift only after significant user harm.

Embedding drift is a leading indicator of coming quality loss. It shows a shift in the input distribution before task metrics move. The right response is to investigate early and refresh the dataset. Waiting for accuracy to drop throws away the point of early-warning monitoring. Rolling back right away is premature, because drift can come from organic changes in user behavior, not just model updates. Raising the alert threshold hides a valid signal and reduces observability.

The quality-evaluation answer is correct. In Langfuse's data model, a Score is an evaluation result (e.g., a relevance rating, a human thumbs-up, or an LLM-as-judge output) attached to a trace or generation to measure quality. It is separate from the generation itself and can be added asynchronously after the fact. The option saying scores store response text and generations store token counts gets both definitions backwards. The option describing a single LLM API call with inputs, outputs, model, and token usage is describing a Generation object, not a Score. The latency-percentile option is wrong because scores are not auto-computed from latency. They are semantic quality evaluations, not performance metrics.