Databricks Data Engineer Associate: Data Ingestion and Loading Practice Questions

Parquet is columnar and compressed, efficient for analytics. CSV, plain text, and fixed-width files (B, C, D) are row-oriented/uncompressed and less efficient.

Trigger(availableNow=True) is the modern recommended approach that processes all available data in multiple micro-batches and then terminates — it combines the 'process all then stop' behavior of Trigger(once=True) but does so more efficiently by splitting work across multiple micro-batches. Trigger(once=True) is the older equivalent that processes everything in a single micro-batch, which can be very slow or cause memory pressure on large datasets and is now considered legacy. Trigger(processingTime='0 seconds') runs micro-batches as fast as possible continuously and never terminates on its own. Trigger(continuous='1 second') enables experimental continuous processing mode with approximately 1-second checkpoint intervals, which does not terminate automatically.

Option E is correct: dlt.read() is a batch/static read that re-reads the entire source table on every pipeline refresh. As the Bronze table accumulates data over time, this full scan becomes increasingly costly in compute and time. Option B is also correct: dlt.read_stream() creates an incremental stream from the source DLT table, reading only records added since the last checkpoint — this is the efficient, scalable approach for Silver tables that process new data continuously. Option C is false — they are fundamentally different in both execution semantics and performance characteristics. Option D is false — dlt.read_stream() does not require a watermark; watermarks are optional and only needed for late-data handling in windowed aggregations. Option F is false — schema evolution behavior in DLT is not differentiated between read() and read_stream(). Option A is false — dlt.read_stream() is the standard way to incrementally read from upstream DLT Streaming Tables, not just external sources.

A Streaming Table in DLT consumes data incrementally using Structured Streaming semantics. By reading from the Bronze source with spark.readStream, the Silver table only processes new records since the last pipeline run, which is the incremental processing behavior required. Option B is incorrect: a Materialized View always re-executes its defining query against the full source on each pipeline update — it does not maintain streaming offsets and therefore cannot do incremental processing in the same sense. Option C is incorrect: creating a table outside DLT with CTAS loses all DLT pipeline management, lineage, and incremental processing capabilities. Option D is a fabrication — DLT Streaming Tables work perfectly well with Delta table sources via spark.readStream; there is no restriction that limits Streaming Tables to Kafka sources.

Setting cloudFiles.schemaEvolutionMode to addNewColumns instructs Auto Loader to automatically detect newly added columns in incoming files and merge them into the target table's schema, allowing the stream to continue without interruption. Option A (rescue) routes unexpected columns to a special _rescued_data column rather than merging them into the schema, which is useful for capturing unexpected data but does not evolve the schema. Option C (inferColumnTypes) controls type inference during initial schema detection, not ongoing schema evolution for new columns. Option D disables automatic evolution entirely and requires manual intervention, which defeats the requirement for automatic handling.