Maintaining and automating workloads Study Guide — Professional Data Engineer

Maintaining and Automating Workloads

This domain tests your ability to keep data pipelines running reliably, efficiently, and cost-effectively on Google Cloud. It covers scheduling, orchestration, monitoring, cost control, error handling, quota management, and deployment automation. At 18% of the exam, it is the second-largest domain and frequently presents scenario-based questions requiring you to choose between overlapping tools (e.g., Cloud Composer vs. Cloud Scheduler vs. Workflows) or diagnose a pipeline problem from a described symptom.

Cloud Composer DAG Scheduling and Dependencies

Cloud Composer is Google's managed Apache Airflow service. It orchestrates complex, multi-step data pipelines using Directed Acyclic Graphs (DAGs) written in Python.

Key Concepts

• A DAG defines tasks and their dependencies. Tasks do not share state; they communicate via XComs or external storage (GCS, BigQuery).
• Operators define what a task does. Exam-relevant operators:
  • BigQueryInsertJobOperator — runs a BQ query or job
  • DataflowCreateJavaJobOperator / BeamRunPythonPipelineOperator — launches Dataflow
  • GCSToGCSOperator, GCSToBigQueryOperator — data movement
  • BashOperator, PythonOperator — general compute
  • ExternalTaskSensor — waits for a task in a different DAG
  • BigQueryCheckOperator / BigQueryValueCheckOperator — data quality gates
• Sensors are a special operator subclass that poll until a condition is true. They consume a worker slot while waiting (use mode="reschedule" to release the slot between polls and avoid deadlocks in small environments).
• Schedule intervals use cron syntax or presets (@daily, @hourly). The start_date is the logical start, not the wall-clock trigger time; Airflow triggers a run at start_date + schedule_interval.
• Catchup: if catchup=True (default), Airflow will back-fill all missed runs since start_date. Set catchup=False to avoid unexpected backfills.

Composer Versions and Environments

Common Traps

• Confusing execution_date (logical date of the interval start) with the actual wall-clock run time.
• Using mode="poke" on sensors in a small environment — workers starve; use mode="reschedule".
• Setting catchup=True accidentally on a DAG with an old start_date — triggers dozens of backfill runs.
• ExternalTaskSensor requires the execution_date to match between DAGs; if schedules differ, you must use execution_delta or execution_date_fn.
• Composer stores DAG files in a GCS bucket; changing a DAG file in the bucket automatically deploys it within ~1–3 minutes.

Worked Scenario

Requirement: A DAG runs daily at 02:00 UTC to load data into BigQuery. A second DAG must run only after the first DAG's load task succeeds. Both DAGs are in the same Composer environment.

Reasoning: Use ExternalTaskSensor in the second DAG pointing to the first DAG's task ID. Set mode="reschedule" to avoid holding a worker slot. Ensure both DAGs share the same schedule_interval and start_date, or configure execution_delta if they differ. Do not use Cloud Scheduler for inter-DAG dependencies — it has no awareness of task status.

Workflows and Cloud Scheduler

These are two distinct services that the exam often conflates.

Cloud Scheduler is a fully managed cron job service. It sends HTTP/Pub/Sub/App Engine messages on a time-based schedule. It has no execution logic — it only triggers.

Google Cloud Workflows is a serverless orchestration engine for chaining Google Cloud API calls and HTTP endpoints using a YAML or JSON workflow definition. It is stateful, supports branching, retries, and parallel steps, but has no built-in cron trigger; it must be triggered by Cloud Scheduler, Pub/Sub, or Eventarc.

Above: Decision tree for choosing between Cloud Composer, Cloud Workflows, and Cloud Scheduler.

Comparison Table

Common Traps

• Cloud Scheduler cannot chain steps or react to success/failure of the triggered job — it fires and forgets.
• Workflows is not a replacement for Composer when you need Airflow-native sensors, complex dependency graphs, or Python-defined logic.
• Cloud Scheduler jobs run at-most-once per schedule tick; if the endpoint is slow, the next run still fires on schedule (no queue buildup).
• Workflows executions are billed per step (first 5,000 internal steps/month free; then $0.01 per 1,000 steps).
• Cloud Scheduler minimum frequency: every 1 minute.