Serving and scaling models Study Guide — Professional Machine Learning Engineer

Serving and Scaling Models

This domain tests your ability to deploy trained models so they serve predictions reliably, cost-effectively, and at scale. It covers the full deployment lifecycle: choosing the right serving infrastructure, configuring Vertex AI endpoints, optimizing models for inference, and managing rollout strategies such as A/B testing and canary deployments. Weighted at 20% of the exam, this is one of the highest-value domains. Expect scenario-based questions that require you to match a business or technical requirement to the correct GCP service, configuration parameter, or trade-off.

Vertex AI Endpoints: Deployment, Traffic Splitting, and Autoscaling

Core Concepts

A Vertex AI Endpoint is a managed HTTPS resource that hosts one or more Model versions and routes prediction requests. The deployment unit is a DeployedModel, which binds a Model resource to an Endpoint with a specific machine type, replica count, and traffic weight.

Key objects and relationships:

• Model – a registered artifact in Vertex AI Model Registry (contains container image + artifact URI)
• Endpoint – the serving surface; has a stable URL and manages routing
• DeployedModel – a model deployed onto an endpoint; carries hardware config and traffic split

Traffic Splitting

An endpoint can host multiple DeployedModels simultaneously. Each DeployedModel is assigned a traffic percentage. All percentages across deployed models on a single endpoint must sum to 100. This enables:

• A/B testing – stable split between two model versions
• Canary rollout – small percentage (e.g., 5%) to a new version before shifting traffic
• Shadow mode – not a native Vertex AI feature; requires custom routing logic

Traffic split is updated via UpdateEndpoint (REST) or update_endpoint() (Python SDK) with zero downtime.

Autoscaling

Setting min_replica_count=0 enables scale-to-zero, which reduces cost when traffic is sporadic but introduces cold-start latency (often 30–120 seconds for large models). For latency-sensitive production workloads, set min_replica_count >= 1.

Dedicated vs. Shared resources:

• Dedicated endpoints use reserved compute; lower latency, higher cost
• Shared resources (default for small models) use multi-tenancy; higher latency variance

Machine Types for Serving

Use n1-standard-* or n1-highmem-* for CPU-only. Add accelerator config (NVIDIA_TESLA_T4, NVIDIA_TESLA_A100) for GPU serving. Machine type is set at DeployedModel creation and cannot be changed in place—you must deploy a new DeployedModel and shift traffic.

Common traps:

• Believing you can update machine type on a live DeployedModel — you must redeploy
• Assuming traffic split can leave a model at 0% traffic while still routing to it — 0% means no traffic; remove it or keep ≥1%
• Confusing the Endpoint URL (stable) with a DeployedModel ID (changes on redeploy)
• Forgetting that min_replica_count=0 introduces cold starts; exam scenarios about SLA violations often point to this setting
• Mixing up Model (registry artifact) and DeployedModel (runtime instance on endpoint)

Worked scenario: A team runs a fraud detection model on Vertex AI. After deploying a retrained model, they want 10% of live traffic on the new version while keeping 90% on the old. They configure traffic split {old_model_id: 90, new_model_id: 10} on the same endpoint and monitor metrics. This is a canary rollout—both models are live, no DNS change needed, and traffic shift requires only an UpdateEndpoint call.

Online vs. Batch Prediction Trade-offs

Bold decision rule: Use online prediction when you need a response within a single user interaction; use batch prediction when you process a large dataset asynchronously and latency per item is not critical.

Batch prediction specifics:

• Triggered via BatchPredictionJob resource in Vertex AI
• Supports JSONL, CSV, TFRecord (Cloud Storage) and BigQuery tables
• Output is written to a destination you specify; job is asynchronous
• Machine type and accelerator configured per job, not per endpoint
• Does not require a deployed endpoint — uses the Model artifact directly

Common traps:

• Routing a high-volume nightly scoring job through an online endpoint — wasteful; use BatchPredictionJob
• Assuming batch prediction needs a running endpoint — it does not; it accesses the Model registry artifact directly
• Confusing Batch Prediction (Vertex AI managed) with running your own Dataflow or Spark job; both are valid patterns but only BatchPredictionJob is "Vertex AI batch prediction"
• Ignoring that batch prediction output can go to BigQuery — exam questions about downstream SQL analysis should trigger this choice

Worked scenario: A retailer wants to score 50 million customer records every night for a recommendation engine. Latency is irrelevant; cost matters. The correct answer is a BatchPredictionJob reading from BigQuery, writing results to BigQuery, running on a schedule (Cloud Scheduler → Vertex AI Pipelines or direct API). An always-on endpoint would incur node costs 24/7 for a job that runs once a night.