Data Engineer Associate · 21% of the exam

Data Ingestion and Loading: free practice questions

5 sample questions from our 64-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Databricks exam.

1. A data engineer is building a DLT pipeline that reads from an external Kafka topic. They declare the ingestion table using `dlt.create_streaming_table()` and use `spark.readStream` to pull from Kafka. Several weeks later, the pipeline is paused for maintenance. When the pipeline restarts, what ensures that the pipeline resumes processing from where it left off rather than reprocessing all Kafka messages from the beginning?

  • A. The DLT pipeline's built-in checkpoint management, which automatically persists read offsets and stream state in the pipeline's storage location✓ Correct
  • B. Kafka's consumer group offset retention, which keeps the last committed offset so any Spark consumer can resume automatically without any Spark-side state
  • C. The Delta transaction log of the target streaming table, which records the last micro-batch ID and is used by Spark to resume the stream
  • D. Setting `startingOffsets` to `latest` in the Kafka read options, which causes the stream to always resume from the newest available message
Explanation

DLT automatically manages checkpoints for all streaming sources in the pipeline. Checkpoint data — including Kafka partition offsets and any aggregation state — is stored in the pipeline's configured storage location. When the pipeline restarts, it reads the checkpoint to resume exactly where it left off. Option B is partially true (Kafka does retain consumer group offsets), but Spark Structured Streaming does NOT rely on Kafka's consumer group offset mechanism for resumption; it uses its own checkpoint files, making B insufficient and misleading. Option C is incorrect: the Delta transaction log tracks table-level write transactions, not Kafka read offsets; the checkpoint directory is separate from the transaction log. Option D is the opposite of what is needed — `startingOffsets=latest` would skip all messages that arrived during the pause, causing data loss.

2. A data engineer is implementing a Structured Streaming job that joins a stream of order events with a static Delta table of product catalog data. The product catalog is updated periodically. Which approach correctly handles this pattern in Spark Structured Streaming?

  • A. Read the product catalog as a static DataFrame using `spark.read` and join it with the stream; the catalog snapshot is loaded once at query start and is not automatically refreshed during the stream's lifetime✓ Correct
  • B. Read the product catalog using `spark.readStream` and perform a stream-stream join; this is required because static DataFrames cannot be joined with streaming DataFrames
  • C. Use `spark.read` with `option('refreshInterval', '5m')` to configure auto-refresh of the static catalog DataFrame every 5 minutes during the stream
  • D. Read the product catalog as a broadcast variable and join it inside a `foreachBatch` function, using `spark.read` to reload the catalog on every micro-batch
Explanation

In Structured Streaming, you CAN join a streaming DataFrame with a static DataFrame read via `spark.read`. The static DataFrame is loaded once when the query starts and acts as a fixed lookup table for the duration of the stream. This is a supported and common pattern. Option B is incorrect — stream-to-static joins are natively supported; a stream-stream join is unnecessary and more complex. Option C describes a non-existent Spark API; there is no `refreshInterval` option for `spark.read` in Structured Streaming. Option D describes a valid workaround using `foreachBatch` to manually reload the catalog on every micro-batch, which does enable periodic refresh, but it is not the standard approach for a simple static join — the question asks which approach 'correctly handles' the pattern, and option A is the standard, correct answer for a static join.

3. A data engineer sets up a Structured Streaming job that reads from a Delta table and writes aggregated results to another Delta table. The engineer wants each micro-batch to be triggered manually and only once, then the stream should terminate automatically. Which trigger setting should the engineer use?

  • A. Trigger(processingTime='0 seconds')
  • B. Trigger(once=True)
  • C. Trigger(continuous='1 second')
  • D. Trigger(availableNow=True)✓ Correct
Explanation

Trigger(availableNow=True) is the modern recommended approach that processes all available data in multiple micro-batches and then terminates — it combines the 'process all then stop' behavior of Trigger(once=True) but does so more efficiently by splitting work across multiple micro-batches. Trigger(once=True) is the older equivalent that processes everything in a single micro-batch, which can be very slow or cause memory pressure on large datasets and is now considered legacy. Trigger(processingTime='0 seconds') runs micro-batches as fast as possible continuously and never terminates on its own. Trigger(continuous='1 second') enables experimental continuous processing mode with approximately 1-second checkpoint intervals, which does not terminate automatically.

4. A data engineer configures a Structured Streaming watermark and windowed aggregation as follows: ```python ( events_stream .withWatermark('event_time', '10 minutes') .groupBy(window('event_time', '5 minutes'), 'device_id') .agg(count('*').alias('event_count')) .writeStream .outputMode('append') .table('device_window_counts') ) ``` A late event arrives with an `event_time` that is 15 minutes behind the current watermark. What happens to this event?

  • A. The event is silently dropped and not included in any window aggregate, because it falls outside the 10-minute watermark threshold✓ Correct
  • B. The event is included in its corresponding 5-minute window aggregate, and the window result is updated and re-emitted in `append` mode
  • C. The event is held in a late-data buffer for up to 5 minutes (the window duration) and then processed once the window closes
  • D. The stream fails with an error because `append` output mode does not support late data and requires `update` mode when a watermark is defined
Explanation

When a watermark is defined with a delay threshold of 10 minutes, Spark drops any event whose `event_time` is earlier than `(max observed event_time) - 10 minutes`. An event arriving 15 minutes behind the current watermark exceeds this threshold and is silently dropped — it will not contribute to any window aggregate. Option B is incorrect: once a window is closed by the watermark, its aggregate is finalized in `append` mode and emitted; the engine will not update and re-emit a finalized window result for a late arrival. Option C describes a fabricated 'late-data buffer' mechanism that does not exist in Structured Streaming — the watermark IS the mechanism for handling late data, and data beyond the threshold is dropped, not buffered. Option D is incorrect: `append` mode IS compatible with windowed aggregations that use a watermark; the watermark is actually required to use `append` mode with aggregations, as it allows Spark to determine when a window result is final.

5. A data engineer defines the following expectation in a DLT pipeline: ```python @dlt.table @dlt.expect_or_drop("valid_amount", "transaction_amount > 0") def silver_transactions(): return dlt.read_stream("bronze_transactions") ``` What happens to records where `transaction_amount` is NULL or 0?

  • A. The records are written to a quarantine table automatically created by DLT for failed expectations.
  • B. The records are dropped from the output table, and the pipeline continues running. The number of dropped records is tracked in the pipeline event log.✓ Correct
  • C. The pipeline run fails immediately when the first record violating the expectation is encountered.
  • D. The records are written to the output table with a warning flag column `_expectation_failed` set to true.
Explanation

**Correct: B.** The `@dlt.expect_or_drop` decorator causes DLT to silently drop any records that violate the specified constraint (where `transaction_amount` is NULL, 0, or negative). The pipeline continues processing without failure, and the dropped record counts are surfaced in the DLT pipeline event log for monitoring. **A** is wrong because DLT does not automatically create a quarantine table; quarantining invalid records requires the engineer to explicitly implement that pattern using `@dlt.expect` and filtering. **C** is wrong because that is the behavior of `@dlt.expect_or_fail`, not `@dlt.expect_or_drop`. **D** is wrong because DLT does not add warning flag columns automatically; that is not a built-in behavior of any DLT expectation decorator.

59 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free