Professional Machine Learning Engineer · 13% of the exam

Monitoring ML solutions: free practice questions

5 sample questions from our 33-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Google Cloud exam.

1. Your team manages a Vertex AI endpoint serving a large language model for document summarization. The endpoint is experiencing latency spikes during peak business hours. You need to ensure the endpoint automatically scales to handle demand while minimizing cost during off-peak hours. What is the MOST appropriate configuration?

A. Set a fixed number of replicas equal to peak demand capacity and use a dedicated endpoint to guarantee latency SLOs
B. Enable autoscaling on the endpoint by configuring minimum and maximum replica counts, and set an appropriate traffic split✓ Correct
C. Deploy the model to a Vertex AI Batch Prediction job scheduled during peak hours to handle the load burst
D. Increase the machine type to the largest available GPU instance to handle all traffic on a single replica

Explanation

Configuring autoscaling with minimum and maximum replica counts is the correct approach for handling variable traffic on a Vertex AI endpoint. It allows the endpoint to scale out during peak hours and scale in during off-peak hours, optimizing both performance and cost. Option A (fixed replicas at peak capacity) ensures availability but wastes resources and cost during off-peak hours—it does not minimize cost. Option C (Batch Prediction) is designed for offline, asynchronous workloads and cannot serve real-time, low-latency online prediction requests. Option D (single large replica) is a fragile design—a single point of failure with no redundancy—and may still not handle peak traffic if a single machine's throughput is insufficient.

2. You are configuring prediction logging for a high-volume Vertex AI endpoint that processes 100,000 requests per hour. You need to log predictions for monitoring model performance and debugging, but storage costs are a concern. You want to maintain statistical validity for performance metrics while minimizing costs. Which approach should you implement?

A. Log every prediction request and response, and use Dataflow to sample 1% of logs for cost reduction.
B. Implement stratified sampling at prediction time to log 10% of requests, ensuring you maintain representative samples across different prediction classes/values, then configure model monitoring on these sampled logs.✓ Correct
C. Log predictions only when confidence scores fall below a threshold (e.g., below 0.7), as these are the most important for troubleshooting.
D. Disable prediction logging entirely and rely solely on Vertex AI Model Monitoring's built-in skew/drift detection metrics.

Explanation

The correct answer is B. Stratified sampling at prediction time (logging 10% of requests) is more cost-effective than post-hoc sampling and maintains statistical validity by ensuring representation across different prediction classes. Vertex AI Model Monitoring can operate on stratified samples. A is incorrect because logging 100% first then sampling wastes storage resources; sampling should happen at prediction time. C is wrong because logging only low-confidence predictions introduces selection bias—you miss high-confidence errors and cannot accurately assess overall model performance. D is incorrect because disabling prediction logging eliminates visibility into prediction quality and would prevent timely detection of performance issues beyond just skew/drift.

3. Your team has deployed a binary classification model on Vertex AI that predicts customer churn. After several weeks in production, you notice that the model's prediction score distribution has shifted significantly compared to the training baseline, but the input feature distributions appear stable. Which type of model degradation is occurring, and what is the most appropriate first action?

A. Feature skew; retrain the model immediately using the latest production data
B. Prediction drift; investigate whether the relationship between features and the target label has changed in the real world✓ Correct
C. Training-serving skew; audit the feature engineering pipeline to ensure consistency between training and serving
D. Data drift; enable continuous evaluation against a held-out validation set

Explanation

When input features are stable but prediction score distributions shift, this is prediction drift (also called concept drift), meaning the underlying relationship between features and the target has changed. The correct first action is to investigate whether real-world dynamics have shifted. Option A is wrong because feature skew refers to discrepancies between training and serving feature values, and immediate retraining without understanding the root cause is premature. Option C is wrong because training-serving skew specifically refers to inconsistencies in how features are computed between training and serving pipelines—not applicable here since features appear stable. Option D is wrong because data drift describes changes in the input feature distribution, which the question states is not occurring here.

4. A team is monitoring costs for a Vertex AI training workload that uses a large cluster of A100 GPUs. They notice that GPU utilization reported in Cloud Monitoring averages only 35% throughout training runs. Which THREE actions would most effectively improve GPU utilization and reduce the cost per training run? (Select THREE)

A. Increase the per-device batch size to better saturate GPU memory and compute✓ Correct
B. Use a tf.data pipeline with prefetching and parallel data loading to eliminate CPU-bound data bottlenecks✓ Correct
C. Enable Vertex AI TensorBoard to visualize training curves and identify convergence issues
D. Profile the training job using Cloud Profiler or TensorBoard's trace viewer to identify bottlenecks in the data pipeline or host-to-device transfers✓ Correct
E. Migrate from synchronous data-parallel training to a larger single-GPU instance to avoid communication overhead
F. Switch to preemptible (Spot) VMs for the training cluster to reduce per-hour costs

Explanation

Low GPU utilization (35%) typically means GPUs are starved for data or compute is bottlenecked elsewhere. Increasing batch size (A) directly increases the amount of work per GPU step, improving utilization. Optimizing the data pipeline with tf.data prefetching and parallel loading (B) eliminates CPU-bound I/O bottlenecks that leave GPUs idle waiting for data. Profiling with Cloud Profiler or TensorBoard's trace viewer (D) is essential to identify the exact bottleneck (e.g., data loading, host-to-device transfers, kernel launch overhead) before applying further fixes. Option C (TensorBoard for training curves) helps with convergence analysis but does not directly address GPU utilization. Option E (migrating to a single GPU) reduces parallelism and would increase training time, not improve efficiency; it may also not be feasible for large models. Option F (Spot VMs) reduces hourly cost but does not improve GPU utilization percentage and introduces the risk of preemption mid-run, which can waste already-consumed compute.

5. Your organization uses a Vertex AI endpoint to rank search results in an e-commerce platform. The ranking model was trained on user click-through data from 2023. You deploy the model in January 2024 and configure Vertex AI Model Monitoring for prediction drift. In March 2024, monitoring alerts show significant prediction drift on the 'user_session_duration' feature. However, business stakeholders report that search result rankings appear reasonable and user engagement metrics (click-through rate, conversion rate) have not degraded. What is the MOST APPROPRIATE interpretation and action?

A. Prediction drift is a leading indicator of upcoming performance problems. Retrain the model immediately as a precaution.
B. The prediction drift alert is likely a false positive due to legitimate seasonal changes in user behavior. Verify that business metrics remain healthy and adjust monitoring thresholds if appropriate. Continue monitoring closely.✓ Correct
C. Prediction drift on one feature indicates the entire model is degrading. Halt the endpoint and investigate.
D. User engagement metrics are the true measure of success. Ignore prediction drift alerts and focus only on business metrics.

Explanation

The correct answer is B. Prediction drift (changes in feature distributions) is not the same as model performance degradation. While drift can be a leading indicator, if business metrics (CTR, conversion) remain strong, the model is still delivering value. The drift is likely due to seasonal user behavior changes that don't harm predictions. The appropriate response is to verify business health, adjust thresholds if they are too sensitive, and continue monitoring. A is incorrect because preemptive retraining based on drift without performance degradation wastes resources. C is incorrect because drift on one feature doesn't warrant shutting down a functioning system. D is overly dismissive—drift should still be monitored, but it must be contextualized against business metrics.

28 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free