Professional Data Engineer · 25% of the exam

Ingesting and processing data: free practice questions

5 sample questions from our 64-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Google Cloud exam.

1. You are troubleshooting a Dataflow streaming pipeline that reads from Pub/Sub and writes to BigQuery using the BigQuery Storage Write API with a 1-second commit interval. The pipeline occasionally produces duplicate rows in BigQuery, even though you've configured exactly-once delivery. Upon investigation, you discover that the Dataflow job is using the default trigger and has not been configured to checkpoint its state. What is the root cause of the duplicates?

  • A. The BigQuery Storage Write API doesn't support exactly-once semantics; you must use streaming inserts instead.
  • B. The 1-second commit interval is too short; increase it to at least 5 seconds to allow Dataflow's state backend to fully synchronize.
  • C. Without checkpointing, Dataflow cannot recover the last committed offset to Pub/Sub when the pipeline restarts, causing messages to be reprocessed and rewritten to BigQuery.✓ Correct
  • D. The default trigger flushes data before the Streaming Engine has finished deduplicating messages in the pipeline's state.
Explanation

The correct answer is option 3. Duplicates occur because Dataflow lacks state checkpoint configuration. When the pipeline restarts or updates, it cannot resume from the last processed Pub/Sub offset, so it reprocesses messages and writes them again to BigQuery. State checkpoints (stored in Cloud Storage or Datastore) are essential for exactly-once semantics in streaming pipelines. Option 1 is incorrect because the BigQuery Storage Write API does support exactly-once with proper configuration. Option 2 is incorrect because the commit interval isn't the root cause—the issue is missing state checkpoints, not commit timing. Option 4 is incorrect because the default trigger and Streaming Engine don't have a built-in deduplication mechanism; deduplication depends on state checkpoints and idempotent writes.

2. A startup is building a real-time data pipeline where a Dataflow streaming job experiences sudden spikes in Pub/Sub message volume (up to 10x normal). The team wants the pipeline to handle these spikes without manual intervention. Which Dataflow feature is MOST relevant?

  • A. Dataflow Shuffle Service, which offloads shuffle operations to a managed backend to reduce worker memory pressure.
  • B. Dataflow horizontal autoscaling, which automatically adds or removes worker VMs based on the backlog in the input source.✓ Correct
  • C. Dataflow FlexRS (Flexible Resource Scheduling), which uses a mix of preemptible and standard VMs to reduce costs during spikes.
  • D. Dataflow templates, which allow the pipeline to be restarted quickly with a larger initial worker count when spikes are detected.
Explanation

Dataflow's horizontal autoscaling continuously monitors pipeline throughput and the unprocessed backlog (e.g., the number of unacknowledged Pub/Sub messages) and automatically scales the number of worker VMs up or down to meet demand, directly addressing sudden traffic spikes without manual intervention. Option A is incorrect because the Shuffle Service (now part of the Streaming Engine for streaming jobs) offloads state and shuffle data but does not by itself scale worker count in response to volume spikes. Option C is incorrect because FlexRS is a batch-only feature that schedules jobs on spare capacity using preemptible VMs to reduce cost; it is not available for streaming jobs and does not address real-time spike handling. Option D is incorrect because Dataflow templates allow parameterized job launches but do not provide automatic scaling—they still require a human or external automation to detect the spike and trigger a restart.

3. Your Dataflow streaming pipeline reads from Pub/Sub, applies a series of transformations, and writes to BigQuery using streaming inserts. During a period of high load, you notice increased latency (messages are delayed by up to 30 seconds from ingestion to BigQuery), and the Dataflow job's worker CPU utilization is at 95%. You've verified that BigQuery is not the bottleneck. What is the most effective immediate mitigation?

  • A. Switch from streaming inserts to the BigQuery Storage Write API with a 30-second batch window to reduce write overhead.
  • B. Increase the Dataflow worker machine type from n1-standard-4 to n1-standard-16, allowing more CPU-intensive transformations per worker.
  • C. Enable Dataflow autoscaling to increase the number of worker VMs, distributing the transformation workload and reducing latency.✓ Correct
  • D. Reduce the transformation complexity by moving some validations to a pre-processing step on Cloud Functions.
Explanation

The correct answer is option 3. Since worker CPU is at 95% utilization (a bottleneck) and BigQuery is not the bottleneck, the issue is transformation throughput. Autoscaling adds more worker VMs to parallelize the transformation work, reducing latency and CPU pressure on individual workers. This is the most direct solution. Option 1 is incorrect because switching to the Storage Write API would actually increase latency (batching for 30 seconds) and doesn't address the CPU bottleneck in transformations. Option 2 is incorrect because upgrading a single worker's machine type doesn't parallelize work if the job is already configured to use all available CPU; autoscaling (adding more workers) is more effective. Option 4 is incorrect because moving transformations to Cloud Functions adds complexity and doesn't integrate well with Dataflow's stateful processing; it's a band-aid solution.

4. A Dataflow streaming pipeline reads from Pub/Sub and writes aggregate counts to BigQuery in 5-minute windows. Analysts report that some late-arriving events (up to 10 minutes late) are being silently dropped. Which Apache Beam concept should be configured to correctly handle these late records?

  • A. Increase the Pub/Sub acknowledgement deadline to 10 minutes so messages are never considered late.
  • B. Set an allowed lateness of 10 minutes on the window and configure a trigger to emit updated panes for late data.✓ Correct
  • C. Use a global window instead of fixed windows so all data is processed together without lateness constraints.
  • D. Enable the Dataflow streaming engine to automatically buffer late messages in persistent state.
Explanation

Apache Beam's 'allowed lateness' setting tells the runner how long after a window closes to continue accepting late elements. Pairing it with an accumulating or accumulating-and-retracting trigger causes Dataflow to emit updated results when late data arrives, rather than dropping it. Option A is incorrect because the Pub/Sub acknowledgement deadline controls how long a subscriber has to ack a message; it has no effect on event-time lateness within windowing logic. Option C is incorrect because a global window eliminates time-based aggregation windows entirely, which would change the business logic and still would not surface per-5-minute aggregates. Option D is incorrect because the Dataflow streaming engine is an execution optimization (eliminating shuffle persistence) and does not automatically handle late data policy—that is still controlled by Beam's windowing configuration.

5. A data engineering team is configuring a Cloud Data Fusion pipeline to join a 100 GB dimension table (stored in Cloud Storage as Parquet) with a 5 TB fact table (stored in BigQuery). The pipeline must complete within 4 hours and minimize Cloud Data Fusion compute costs. Which approach is most cost-effective?

  • A. Load the dimension table into BigQuery, then use BigQuery's native JOIN operation within Cloud Data Fusion, avoiding the need for a separate compute cluster.✓ Correct
  • B. Configure Cloud Data Fusion to read the dimension table directly from Cloud Storage and the fact table from BigQuery, letting the Cloud Data Fusion Spark cluster perform the join.
  • C. Export the fact table from BigQuery to Cloud Storage in Parquet format, then use Cloud Data Fusion to join both Parquet files locally.
  • D. Use a lightweight Cloud Data Fusion cluster (2 nodes) and configure it to cache the 100 GB dimension table in memory, preventing repeated disk I/O during the join.
Explanation

The correct answer is option 1. Loading the 100 GB dimension table into BigQuery (which is fast and inexpensive) and performing a native BigQuery JOIN is the most cost-effective approach because: (1) BigQuery's JOIN is optimized for large datasets, (2) you avoid spinning up a large Spark cluster in Cloud Data Fusion, and (3) Cloud Data Fusion can efficiently pull the joined result. Option 2 is incorrect because it requires a full Cloud Data Fusion cluster to perform the join, which is expensive for a relatively small dimension table (100 GB). Option 3 is incorrect because exporting 5 TB from BigQuery to Cloud Storage is costly and slow, defeating the purpose of using BigQuery. Option 4 is incorrect because caching 100 GB in memory on a 2-node cluster is not feasible (each node wouldn't have enough memory), and it doesn't address the fact that the 5 TB fact table still needs to be processed.

59 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free