Google Cloud Professional Data Engineer: Ingesting and processing data Practice Questions

The correct answers are options 1, 3, and 5. Option 1 is correct because the Streaming Engine is essential for maintaining state consistency and exactly-once semantics. Option 3 is correct because slot reservations guarantee query performance for analytics workloads, and the Storage Write API is the recommended approach for efficient, deduplicated writes to BigQuery. Option 5 is correct because dead-letter topic handling enables reliable error recovery and observability—critical for production pipelines. Option 2 is incorrect because using preemptible VMs in a critical real-time pipeline introduces unnecessary disruption risk; standard VMs are more appropriate. Option 4 is incorrect because a 10-second commit interval is arbitrary; the optimal interval depends on latency requirements and write volume, and the Storage Write API automatically handles efficient batching. Option 6 is partially correct (autoscaling is good), but it's less critical than the other three considerations for production readiness.

Option 0 is correct. The BigQuery Storage Write API has quota limits, and traffic surges can exceed available write slots. Increasing BigQuery reservation capacity and enabling Dataflow autoscaling (which scales workers to handle increased data volume) directly address the bottleneck. Option 1 is incorrect—maxExtendedDeadlineSeconds relates to message acknowledgment deadlines, not write failures. Option 2 misdiagnoses the issue; write latency to BigQuery is rarely CPU-bound, and changing machine type without addressing the actual bottleneck (BigQuery Write API quota) won't help. Option 3 is incorrect—partitioning and batch size optimization are valid tuning techniques, but they don't address quota limits or autoscaling failures.

The correct answer is option 3. Since worker CPU is at 95% utilization (a bottleneck) and BigQuery is not the bottleneck, the issue is transformation throughput. Autoscaling adds more worker VMs to parallelize the transformation work, reducing latency and CPU pressure on individual workers. This is the most direct solution. Option 1 is incorrect because switching to the Storage Write API would actually increase latency (batching for 30 seconds) and doesn't address the CPU bottleneck in transformations. Option 2 is incorrect because upgrading a single worker's machine type doesn't parallelize work if the job is already configured to use all available CPU; autoscaling (adding more workers) is more effective. Option 4 is incorrect because moving transformations to Cloud Functions adds complexity and doesn't integrate well with Dataflow's stateful processing; it's a band-aid solution.

Option 0 is correct. Throughput-based autoscaling automatically scales workers based on the rate of data processing, which is the appropriate metric for batch jobs. A target utilization of 70% balances cost and performance. Option 1 is incorrect because the Streaming Engine is designed for streaming workloads, not batch jobs, and the option explicitly disables batch jobs. Option 2 is wrong because manual fixed worker counts don't adapt to actual workload demands and can result in wasted resources or insufficient capacity. Option 3 is partially correct about Shuffle service and batch jobs, but sets maxWorkers to only 20, which may be insufficient for the 3x speedup needed (8TB in 2 hours vs. 6 hours).

Option 1 is correct. Pub/Sub provides at-least-once semantics by design (no exactly-once guarantee). Dataflow's deduplication (storing message IDs in state) combined with BigQuery's MERGE statement (idempotent upsert) ensures end-to-end exactly-once semantics. This is the standard pattern for exactly-once financial transactions. Option 0 is wrong because Pub/Sub does not offer exactly-once delivery; dead-letter topics are for handling failed messages, not deduplication. Option 2 is incorrect—Dataflow does not have an 'exactly-once processing mode' in this context; it only guarantees exactly-once state updates. Option 3 is wrong because BigQuery does not provide end-to-end exactly-once guarantees across a pipeline; that responsibility falls to the pipeline orchestration.