AI Advanced · 20% of the exam

AI Observability & Evals: free practice questions

5 sample questions from our 160-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real DataCertPrep exam.

1. A team is designing an LLM observability strategy and debates whether to use OpenTelemetry (OTel) directly versus a purpose-built tool like Langfuse or Braintrust. Which statement MOST accurately describes the relationship between OpenTelemetry and LLM-specific observability platforms?

A. OpenTelemetry and LLM observability platforms are mutually exclusive; you must choose one architecture.
B. OpenTelemetry provides the general-purpose instrumentation and data transport layer (traces, spans, attributes), while LLM-specific platforms add semantic conventions, token/cost analytics, evaluation scoring, and AI-focused UI — they are complementary, not competing.✓ Correct
C. LLM observability platforms like Langfuse internally replace OpenTelemetry with a proprietary protocol to reduce latency overhead.
D. OpenTelemetry's GenAI semantic conventions make purpose-built LLM observability platforms obsolete for production use cases.

Explanation

Option B is correct. OpenTelemetry is an open standard for telemetry data collection and export, and LLM platforms like Langfuse can ingest OTel traces or provide their own SDKs. They are complementary: OTel gives you vendor-neutral instrumentation, while LLM platforms give you AI-specific analysis, scoring, and dashboards on top of that telemetry. Option A is false; many teams use both in their stack. Option C is false; Langfuse supports OTel ingestion and does not replace it with a proprietary protocol. Option D overstates OTel's current scope — the GenAI semantic conventions standardize attribute names but do not provide evaluation scoring, human feedback collection, or experiment comparison.

2. A product team is running an A/B test between two system prompts for a writing assistant. After one week, Prompt A has 1,200 sessions and a mean helpfulness score of 0.82 (σ = 0.15), while Prompt B has 1,180 sessions and a mean helpfulness score of 0.85 (σ = 0.14). The p-value is 0.003. The team is ready to ship Prompt B. Which concern, if overlooked, would MOST undermine the validity of this conclusion?

A. The standard deviations are different between the two groups, which invalidates any statistical test.
B. The experiment may have suffered from novelty effect bias if users who received Prompt B were heavier users who also happened to give higher scores regardless of prompt quality.✓ Correct
C. A p-value of 0.003 is below the conventional 0.05 threshold but above the stricter 0.001 threshold, so the result is not significant.
D. The sample sizes are unequal (1,200 vs. 1,180), which means the comparison is statistically invalid.

Explanation

The correct answer is B. Even with a statistically significant result, a confound like user selection bias (e.g., Prompt B was shown to a subpopulation of more engaged or more generous raters) can produce a spurious signal. This is a classic internal validity threat in online experiments. A is incorrect — having slightly different standard deviations between groups does not invalidate a t-test or similar; it may affect which variant of the test is used (Welch's vs. Student's), but neither σ here is dramatically different. C is incorrect — p = 0.003 is well below the conventional α = 0.05 threshold and is considered significant; there is no universal 'stricter' threshold that overrides this. D is incorrect — a difference of 20 samples out of ~1,200 is trivially small and does not invalidate the comparison; unequal sample sizes are handled normally by standard statistical tests.

3. A machine learning engineer notices that an LLM-powered customer support chatbot is producing subtly lower-quality responses over several weeks, but no errors appear in the application logs and user-reported complaints remain low. Which characteristic of AI systems most directly explains why traditional monitoring missed this regression?

A. LLM APIs throttle requests silently, causing incomplete responses that look valid.
B. AI systems exhibit non-determinism and can experience prompt drift, causing quality to degrade without triggering hard errors.✓ Correct
C. Traditional monitoring tools do not support HTTP/2, which modern LLM APIs require.
D. Model weights are updated server-side by providers without notice, invalidating all cached responses.

Explanation

Correct: AI systems are non-deterministic and susceptible to prompt drift — subtle shifts in model behavior, context window handling, or prompt interpretation that degrade output quality without raising exceptions or HTTP error codes. This is the canonical 'silent failure' scenario that motivates AI observability. | Wrong (A): API throttling produces rate-limit errors (HTTP 429) or truncated responses that are observable; it is not a silent quality regression. | Wrong (C): HTTP/2 support is unrelated to quality monitoring; this is a fabricated distractor. | Wrong (D): While providers do update models, this is one possible cause of drift, not an explanation of why traditional monitoring fails — the root cause of the monitoring gap is the lack of quality signals, not the model update itself.

4. A team is instrumenting a multi-step LLM pipeline using OpenTelemetry. The pipeline consists of: (1) a query rewriting LLM call, (2) a vector DB retrieval step, (3) a response generation LLM call. They want end-to-end trace context to propagate correctly so that all three steps appear as child spans under a single root trace in their backend. Which approach correctly achieves this?

A. Create a root span at the entry point of the pipeline, pass the OpenTelemetry context object (containing the trace ID and parent span ID) to each subsequent step, and start child spans within that context for each of the three sub-steps.✓ Correct
B. Create three independent traces — one per step — and link them post-hoc in the observability dashboard using a shared correlation ID stored in the request body.
C. Use OpenTelemetry baggage to store the full prompt text and propagate it through all spans so the backend can reconstruct the trace.
D. Set the same static trace ID hardcoded in all spans; OpenTelemetry will automatically detect identical trace IDs and nest the spans correctly.

Explanation

Correct (A): OpenTelemetry context propagation works by passing the active context (which carries the trace ID and current span ID as the parent) through the call chain. Each child span is started within this context, causing it to appear nested under the root span in the backend. Wrong (B): Creating three independent traces produces three disconnected entries; post-hoc linking via a custom correlation ID is a workaround that loses native parent-child relationships and is not standard OTel practice. Wrong (C): Baggage is designed to carry user-defined key-value data across process boundaries (e.g., feature flags), not to reconstruct trace hierarchies; prompt text in baggage does not establish span parent-child relationships. Wrong (D): Hardcoding a static trace ID is both incorrect (trace IDs must be unique per request, not static) and dangerous — all requests would collide under the same trace ID. OTel does not auto-nest spans based on identical IDs alone.

5. An engineering team runs weekly Braintrust eval experiments on their LLM application and tracks a composite score combining 'accuracy', 'groundedness', and 'tone' sub-scores. Over four consecutive weeks, the composite score remains flat at 0.81. A staff engineer argues this is misleading and requests a change to the reporting approach. Which analysis would MOST effectively reveal whether the flat composite score is masking meaningful model behavior changes?

A. Re-run the eval with a higher temperature setting to increase output diversity and check whether the composite score variance increases.
B. Decompose the composite score into its individual sub-score trends over the four weeks and compare score distributions per dataset slice (e.g., by query type or difficulty tier) to detect compensating shifts.✓ Correct
C. Switch from Braintrust to a different eval platform, as flat scores over four weeks indicate the scoring functions are saturated and need recalibration.
D. Increase the golden dataset size to 10,000 examples so that statistical noise in the composite score is reduced and true trends become visible.

Explanation

Correct: A flat composite score can hide compensating changes — for example, accuracy improving while groundedness deteriorates, or performance improving on easy queries but worsening on hard ones. Decomposing into sub-score trends and slicing by dataset characteristics (query type, difficulty) reveals these offsetting dynamics that aggregate masking obscures. This is the core analytical skill for production LLM observability. — Wrong A: Increasing temperature changes what the model outputs and confounds the evaluation; it tests a different production configuration, not the one being monitored, and adds variance rather than insight. — Wrong C: A flat score over four weeks does not indicate platform saturation; it may simply mean the model is stable. Switching platforms abandons historical trend data without addressing the analytical gap. — Wrong D: Increasing dataset size reduces sampling noise but does not address compensating sub-score shifts; a larger dataset of the same flat composite still wouldn't reveal the decomposed trends.

155 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free