Ingesting and processing data Study Guide — Professional Data Engineer

Ingesting and Processing Data

This domain covers building, deploying, and operating both batch and streaming data pipelines on Google Cloud. It accounts for 25% of the exam, making it the largest single domain. Questions test your ability to choose the right ingestion service, configure pipelines correctly, handle late or out-of-order data, and operationalize pipelines at scale. Expect scenario-based questions where multiple services seem plausible — the exam tests why you pick one over another, not just what each service does.

Pub/Sub: Ordering, Exactly-Once, Dead-Letter Topics, Push vs Pull

Core Concepts

Cloud Pub/Sub is a fully managed, serverless messaging service for asynchronous, durable event streaming. It decouples producers (publishers) from consumers (subscribers).

Key vocabulary:

• Topic: the named resource to which publishers send messages.
• Subscription: a named resource representing a stream of messages from a single topic.
• Message: data plus optional attributes; max size 10 MB.
• Acknowledgment deadline: how long a subscriber has to ack a message before it is redelivered; default 10 seconds, configurable up to 600 seconds.
• Message retention: up to 7 days (default 10 minutes for undelivered messages after subscription creation; topic retention up to 31 days when enabled).

Delivery Semantics

Ordering keys: when a publisher sets an ordering key on a message, Pub/Sub guarantees ordered delivery within the same region and within the same ordering key. If one message in an ordering-key sequence fails, subsequent messages with the same key are paused until the failure is resolved or the subscriber calls modifyAckDeadline.

Dead-Letter Topics

A dead-letter topic (DLT) is a separate topic to which Pub/Sub automatically forwards messages that exceed a configurable maximum delivery attempt count (5–100 attempts, default 5). Use a DLT to:

• Inspect messages that cannot be processed.
• Prevent a single bad message from blocking a pipeline.

The DLT requires the Pub/Sub service account to have Publisher role on the dead-letter topic and Subscriber role on the original subscription.

Push vs Pull

Common Traps

• Exactly-once is not free: enabling it increases latency and cost; do not recommend it when idempotent consumers are an option.
• Ordering keys do not work across regions: if a publisher publishes to multiple regions, ordering is not preserved globally.
• DLT does not automatically retry: once in the DLT, you must build a separate consumer to reprocess.
• Pull subscription message ordering: even with ordering enabled, a single subscriber client must be used per ordering key, or ordering guarantees break.
• Snapshot and seek: Pub/Sub supports seek to replay messages to a past timestamp or snapshot — a common distractor presented as "only Kafka can do this."

Dataflow (Apache Beam): Windowing, Watermarks, Triggers, Late Data

Apache Beam Programming Model

Cloud Dataflow is the fully managed execution engine for Apache Beam pipelines. Beam is the SDK (Java, Python, Go); Dataflow is the runner.

PCollection: the core data abstraction — an immutable, distributed dataset. Can be bounded (batch) or unbounded (streaming).

Windowing

Windowing divides an unbounded stream into finite chunks for aggregation.

Watermarks

A watermark is an estimate of how far behind real time the pipeline's event-time processing is. It represents a threshold: "we believe all events with timestamps before W have arrived."

• Event time: when the event actually occurred (embedded in the message).
• Processing time: when the pipeline processes the message.
• Skew: the difference between event time and processing time.

When the watermark passes the end of a window, Dataflow emits the window's result. Dataflow infers watermarks automatically from Pub/Sub message timestamps.

Triggers

Triggers control when window results are emitted.

Accumulation modes:

• Discarding: each pane contains only new data since the last trigger.
• Accumulating: each pane contains all data seen so far (may cause double-counting downstream).

Late Data

Allowed lateness (withAllowedLateness) tells Dataflow how long after the watermark to continue accepting late-arriving data. After this duration, late data is dropped (or forwarded to a side output).

Side outputs (tagged outputs) allow late data to be routed to a separate PCollection for separate handling rather than being silently dropped.

Pipeline decision flow:

Diagram 1 — Choosing the correct Dataflow windowing and trigger strategy

Common Traps

• Watermarks estimate, not guarantee: data later than the allowed lateness is dropped — exam questions test whether you handle this with side outputs.
• Accumulating mode causes duplicates if the downstream system is not idempotent — a frequent wrong-answer pattern.
• Session windows require per-key processing: they cannot be used without a GroupByKey or equivalent.
• Global windows with default trigger never fire on streaming data — you must set a custom trigger.
• AfterWatermark is NOT real-time: for low-latency speculative results, combine with AfterProcessingTime using AfterAny.