Monitoring ML Solutions
This domain covers the tools, techniques, and workflows used to keep deployed ML models performing correctly and cost-effectively over time. It accounts for 13% of the exam, making it one of the mid-weight domains. Questions test your ability to select the right monitoring configuration, interpret alerts, diagnose failures, and decide when and how to retrigger training — using GCP-native services such as Vertex AI Model Monitoring, Vertex Explainable AI, Cloud Logging, and Cloud Monitoring.
Vertex AI Model Monitoring: Skew and Drift Detection
Core Concepts
Training-serving skew occurs when the feature distribution seen at serving time differs from the distribution in the training dataset. Feature drift (also called prediction drift) occurs when the serving distribution shifts over time, even without a reference to training data.
Vertex AI Model Monitoring supports both detection types for deployed Vertex AI Endpoints. It works by sampling a configurable percentage of prediction requests, storing them in a BigQuery table, and comparing feature distributions against a baseline using statistical tests.
| Concept | Baseline | Statistical Test | Trigger |
|---|---|---|---|
| Training-serving skew | Training dataset (BigQuery or GCS) | Jensen-Shannon divergence (categorical), z-score/mean-std (numerical) | Feature distribution in requests differs from training data |
| Feature/prediction drift | Recent past serving window | Jensen-Shannon divergence (categorical), z-score (numerical) | Feature distribution changes over time |
| Prediction drift | Recent past window | Same as feature drift | Model output distribution shifts over time |
Configuration Details
- Sampling rate: 1%–100% of traffic; default is not fixed — you set it explicitly in the monitoring job config.
- Monitoring frequency: minimum 1 hour, configurable up to days.
- Alert threshold: You set per-feature thresholds for the divergence score. Exceeding the threshold generates an email alert and a log entry in Cloud Logging.
- Supported model types: tabular models deployed on Vertex AI Endpoints. Custom containers and non-tabular models require additional instrumentation.
- Training dataset requirement for skew: must provide schema + statistics from the training dataset (as a BigQuery table URI or a GCS path to a TFRecord/CSV).
Diagram: Data flow for Vertex AI Model Monitoring skew and drift detection.
Common Traps
- The exam may present data drift and concept drift as if Vertex AI Model Monitoring detects concept drift — it does not. It detects feature/prediction distribution drift, not changes in the underlying relationship between features and labels.
- Skew detection requires a training dataset reference. If no baseline is provided, only drift detection is possible.
- Model Monitoring is scoped to Vertex AI Endpoints — it does not natively monitor models served via Cloud Run, GKE, or other custom serving infrastructure.
- Alerts go to email only (configured at job creation) plus Cloud Logging — not natively to PagerDuty or Slack without additional integration.
- Monitoring jobs must be explicitly created; deploying a model to an endpoint does NOT automatically enable monitoring.
Worked Scenario
Requirement: A team trains a fraud detection model on last year's transaction data and serves it on a Vertex AI Endpoint. They want to know if live transactions start looking different from the training distribution.
Reasoning: This is a training-serving skew detection use case. The team needs to configure a Vertex AI Model Monitoring job, provide the training BigQuery table as the baseline, set a sampling rate, and configure per-feature divergence thresholds. Drift detection alone would not compare against training data.
Prediction Logging and Alerting
Prediction Logging
Vertex AI Endpoints support two types of logging:
| Log Type | What is Recorded | Where Stored | Default State |
|---|---|---|---|
| Request/Response logging | Full input features and prediction outputs | BigQuery table (user-specified) | Disabled |
| Access logging | HTTP metadata (latency, status codes, endpoint ID) | Cloud Logging | Enabled by default |
| Container logging | stdout/stderr from the serving container | Cloud Logging | Enabled by default |
- Enable request/response logging by setting
enable_request_response_logging: trueand specifying abigquery_destinationwhen creating or updating the endpoint. - Sampling rate for request/response logging is configurable (0.0–1.0).
- BigQuery destination must be in the same project (cross-project logging is not supported natively).
Alerting with Cloud Monitoring
Vertex AI surfaces metrics to Cloud Monitoring under the aiplatform.googleapis.com resource type. Key metrics:
| Metric | Description |
|---|---|
prediction/latency | Per-request prediction latency |
prediction/online_prediction_requests_count | Total requests per endpoint |
prediction/online_prediction_error_count | HTTP error count per endpoint |
prediction/model_server_cpu | CPU utilization of model server |
- Create alerting policies in Cloud Monitoring on these metrics with threshold conditions.
- Use notification channels (email, PagerDuty, Slack via webhook, PubSub) for routing.
- Log-based alerts can be created on Cloud Logging for specific error patterns (e.g., 5xx errors, OOM messages).
Common Traps
- Students confuse access logging (always on, metadata only) with request/response logging (off by default, captures features and predictions).
- Request/response logging goes to BigQuery, not Cloud Logging — a common distractor.
- Model Monitoring and request/response logging are separate features even though both use BigQuery; enabling one does not enable the other.
- Log-based metrics in Cloud Monitoring require first creating the metric from a log query before building an alert on it.
Unlock with a paid plan
7 more sections (about 2,320 words) continue below. Every paid plan unlocks all Professional Machine Learning Engineer study guides, the full question bank and mock exams, from $4.97/month.