Automating and orchestrating ML pipelines Study Guide — Professional Machine Learning Engineer

Automating and Orchestrating ML Pipelines

This domain tests your ability to design, build, and maintain production-grade ML pipelines on Google Cloud using automation, CI/CD principles, and orchestration tools. It accounts for 22% of the exam — the single largest domain — meaning roughly 13–15 questions on a 60-question exam. Questions test both conceptual understanding (when to use which tool) and implementation detail (how components, triggers, and registries work together). Expect scenario-based questions that require you to choose between Vertex AI Pipelines, Cloud Composer, and manual approaches, and to reason about versioning, lineage, and deployment gates.

Vertex AI Pipelines: Kubeflow Pipelines and TFX

Core Concepts

Vertex AI Pipelines is the managed pipeline orchestration service on Google Cloud. It runs pipelines defined using either the Kubeflow Pipelines (KFP) SDK v2 or TensorFlow Extended (TFX). Both compile to the same Intermediate Representation (IR YAML) format and execute as directed acyclic graphs (DAGs) on managed infrastructure — you do not provision or manage the underlying compute.

Components and DAGs

A pipeline component is the smallest unit of work. In KFP v2, you create components by decorating Python functions with @component or by building a container-based component that specifies a base image and command. Components communicate by passing artifacts (large data, stored in Cloud Storage) and parameters (small values, stored in pipeline metadata).

A DAG (Directed Acyclic Graph) is the pipeline structure. Dependencies between components are declared implicitly by passing one component's output as another component's input — KFP resolves execution order automatically. There are no cycles allowed.

TFX standard components you must know for the exam:

• ExampleGen — ingests data, creates train/eval splits
• StatisticsGen — computes statistics over data
• SchemaGen — infers schema from statistics
• ExampleValidator — detects anomalies against schema
• Transform — feature engineering with tf.Transform
• Trainer — trains the model
• Evaluator — evaluates against a baseline with TFMA; produces blessing
• Pusher — pushes blessed model to serving location or Model Registry

The blessing pattern: Evaluator compares the new model to a baseline. If the new model meets thresholds, it outputs a ModelBlessing artifact. Pusher only proceeds if the model is blessed. This is a critical exam concept for automated validation gates.

TFX pipeline DAG showing the blessing gate that prevents unqualified model deployment.

Common Traps

• KFP v1 vs v2: The exam uses v2 (IR YAML). KFP v1 used Argo Workflows directly; v2 abstracts this. Don't apply v1-specific syntax reasoning.
• Artifacts vs Parameters: Large data (datasets, models) = artifacts stored in GCS. Small values (learning rate, accuracy number) = parameters in Metadata Store. Confusing these leads to wrong answers on data passing questions.
• TFX on Vertex vs on-prem: On Vertex, TFX pipelines use Vertex AI Pipelines as the runner. The BeamDagRunner is for local/Dataflow, not Vertex.
• Vertex AI Pipelines is not Kubeflow on GKE: Vertex AI Pipelines is serverless. You do not manage Kubernetes. Kubeflow on GKE is a separate, self-managed option the exam uses as a distractor.

Pipeline Triggers and Scheduling

Triggering Mechanisms

Vertex AI Pipelines can be triggered in three primary ways, each serving different automation needs:

Scheduling with Vertex AI Pipelines

Vertex AI Pipelines has a native scheduling feature (introduced as a first-class feature). You define a PipelineJob with a schedule using cron syntax. Key properties:

• max_concurrent_run_count: limits how many runs execute simultaneously (default 1, max 10).
• start_time / end_time: optional window for the schedule.
• Schedules are managed as Schedule resources under the Vertex AI API.

Cloud Scheduler Integration

For more control or to integrate with other GCP services, Cloud Scheduler sends an HTTP request (or Pub/Sub message) that triggers a Cloud Function or Cloud Run service, which then calls the Vertex AI Pipelines REST API to submit a PipelineJob. This pattern is common in exam scenarios requiring cross-service coordination.

Event-Driven Patterns

When data arrives in Cloud Storage, a Pub/Sub notification can be configured on the bucket. A Cloud Function (or Eventarc trigger) receives the notification and calls aiplatform.PipelineJob(...).submit(). The exam tests whether you know this chain: GCS → Pub/Sub → Eventarc/Cloud Functions → Vertex AI Pipelines API.

Common Traps

• Native scheduling vs Cloud Scheduler: If the question asks for the simplest scheduling solution for Vertex AI Pipelines, the answer is native Vertex AI Pipelines scheduling, not Cloud Scheduler. Cloud Scheduler is correct when you need cross-service coordination or Pub/Sub fanout.
• Pub/Sub alone cannot trigger a pipeline: Pub/Sub delivers messages; you still need a subscriber (Cloud Function, Cloud Run, or Eventarc) to call the Vertex API.
• max_concurrent_run_count default is 1: If a scheduled trigger fires but a run is already in progress, subsequent runs are queued, not dropped (up to the limit). Know this for questions about missed runs.