Data Engineering with Azure Databricks · 33% of the exam

Deploy and maintain data pipelines and workloads: free practice questions

5 sample questions from our 60-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Microsoft exam.

1. A data platform team is implementing a testing strategy for a new Databricks pipeline. A junior engineer proposes running only end-to-end (E2E) tests in production after each deployment to verify correctness. A senior engineer argues this approach is insufficient. Which statement BEST explains why relying solely on E2E tests in production is problematic?

  • A. End-to-end tests cannot be run in Databricks; they must be run in a separate testing framework outside the platform.
  • B. End-to-end tests validate the full pipeline but are slow, expensive to run, and provide little guidance on which specific component failed when an issue is detected; earlier-stage unit and integration tests catch bugs faster and at lower cost.✓ Correct
  • C. End-to-end tests are only valid for streaming pipelines; batch pipelines require unit tests exclusively.
  • D. Running tests in production is acceptable as long as the tests use a separate catalog or schema to isolate test data from production data.
Explanation

E2E tests validate that the entire pipeline produces correct results but are costly to run (require full-scale infrastructure and data), slow to execute, and provide poor fault localization — when they fail, the engineer must investigate the entire pipeline to find the root cause. A layered testing strategy (unit tests → integration tests → E2E → UAT) catches bugs early when they are cheapest to fix. Option A is incorrect because E2E tests can absolutely be run in Databricks; the platform supports various testing frameworks. Option C is incorrect because the streaming vs. batch distinction does not determine which test types are applicable; both pipeline types benefit from all levels of testing. Option D is incorrect because while test isolation using separate schemas is a good practice, running all tests only in production (even in isolated schemas) still exposes the production environment to risk and does not address the fundamental inadequacy of relying solely on E2E tests.

2. A data engineer is building a Databricks pipeline to process clickstream data. The pipeline must enforce schema validation, track data lineage across tables, and allow incremental processing with automatic dependency management between datasets. The engineer is deciding between Databricks notebooks orchestrated via a Lakeflow Job and Databricks Declarative Pipelines (DLT). Which capability is EXCLUSIVELY available in Declarative Pipelines and cannot be replicated with notebooks alone in a Lakeflow Job?

  • A. Scheduling pipelines to run on a recurring cron schedule
  • B. Automatic lineage tracking and dependency resolution between datasets defined in the pipeline✓ Correct
  • C. Reading data incrementally from a cloud storage source using Auto Loader
  • D. Sending email alerts when a pipeline run fails
Explanation

Declarative Pipelines (DLT) automatically infer and manage dependencies between datasets defined as pipeline tables, track data lineage, and handle incremental processing without the engineer manually wiring task dependencies. This is exclusive to the DLT framework. Cron scheduling is available to both DLT pipelines and standard Lakeflow Jobs. Auto Loader can be used in standard notebook-based jobs as well as DLT. Email alerts on failure are supported by standard Lakeflow Job alerting configurations.

3. A data engineer is investigating a slow Spark stage in the Databricks Spark UI. The stage summary shows: median task duration = 4 seconds, maximum task duration = 87 seconds, and 'Input Size / Records' is heavily skewed across tasks. The 'Shuffle Read Blocked Time' metric is near zero. Which root cause and remediation BEST match these observations?

  • A. The cluster has insufficient memory, causing disk spill; increase executor memory or switch to a memory-optimized instance type.
  • B. The shuffle partition count is too high, causing many tiny tasks; decrease `spark.sql.shuffle.partitions`.
  • C. Data skew in the shuffle keys is causing a small number of partitions to receive disproportionately large amounts of data; apply salting or use Adaptive Query Execution (AQE) skew join optimization.✓ Correct
  • D. The Delta table lacks Z-ordering on the join key, causing excessive file scanning; run OPTIMIZE with ZORDER on the join column.
Explanation

Option C is correct. The pattern — median task duration much lower than max task duration, combined with skewed 'Input Size / Records' — is the classic signature of data skew. A small number of tasks receive far more data than others, causing stragglers. The fix is salting the skewed key or enabling AQE's skew join optimization (`spark.sql.adaptive.skewJoin.enabled=true`). Option A is incorrect because disk spill would appear as high 'Shuffle Spill (Memory)' and 'Shuffle Spill (Disk)' metrics, not as skewed input sizes and task duration imbalance alone. Option B is incorrect because too many small shuffle partitions would cause uniformly short task durations, not a large gap between median and maximum. Option D is incorrect because ZORDER affects file pruning during reads (reducing data scanned) but does not address partition-level key skew during shuffle stages.

4. A company streams IoT sensor data into a Delta Lake table using a Databricks Lakeflow Job with automatic retries set to 3. The job fails on attempt 2 due to a transient network timeout connecting to the source system. On attempt 3, the job succeeds and writes data to the Delta table. Later, the data team discovers duplicate records in the table. What is the MOST likely root cause and the recommended prevention strategy?

  • A. The Delta table's transaction log was corrupted during the failed attempt; run FSCK REPAIR TABLE to remove duplicate entries.
  • B. The job's retry logic re-processed records that were partially written during the failed attempt because the write operation was not idempotent; use MERGE INTO with a unique key or enable idempotent writes using a unique transaction ID per batch.✓ Correct
  • C. Delta Lake does not support streaming writes with retries; the team should switch to batch mode to avoid duplicates.
  • D. The duplicate records were caused by a skewed partition that was written twice; run OPTIMIZE with ZORDER to deduplicate the data.
Explanation

When a Spark write partially succeeds before a failure and is then retried, records already written in the partial attempt may be written again on the successful retry, causing duplicates. The fix is to make writes idempotent: use MERGE INTO with a unique business key (upsert semantics) so duplicate source records are matched to existing rows and updated rather than inserted again, or use Structured Streaming's built-in exactly-once guarantees with a unique query ID per batch. Option A is wrong because Delta Lake's ACID transaction log prevents corruption from failed writes — a failed write either commits fully or not at all at the transaction level; however, a retry can re-submit the same records as a new, valid transaction, which Delta's log allows. FSCK does not deduplicate data. Option C is wrong because Delta Lake fully supports streaming writes with retries; this is one of its core use cases. Option D is wrong because OPTIMIZE/ZORDER is a file compaction and layout optimization — it does not deduplicate records and has no awareness of business-level duplicates.

5. A nightly Delta maintenance job runs `OPTIMIZE` followed by `VACUUM` on a large Delta table. After several weeks, the data engineering team notices that time travel queries for dates more than 5 days ago are failing with 'FileNotFoundException'. What is the MOST likely cause?

  • A. The `OPTIMIZE` command is rewriting files and updating the Delta log in a way that invalidates time travel metadata.
  • B. The `VACUUM` command is being run with a retention period shorter than the time travel window that downstream consumers require (e.g., RETAIN 5 DAYS or less, while users need to query 7+ days back).✓ Correct
  • C. The nightly `OPTIMIZE` is generating too many new file versions, exhausting the Delta log storage quota.
  • D. Delta time travel is only supported for 30 snapshots, not 30 days; the table has exceeded its snapshot limit.
Explanation

Option B is correct. VACUUM with a retention period shorter than the consumers' time travel window will delete the data files backing older snapshots. If VACUUM runs with `RETAIN 5 DAYS` but users query 7 days back, those older data files no longer exist, causing FileNotFoundException. The fix is to set the retention period to at least as long as the required time travel window. Option A is incorrect because OPTIMIZE rewrites files and adds new entries to the Delta log, but old file versions remain in the log and on storage; it does not by itself delete old files or break time travel. Option C is incorrect because Delta log files are relatively small and there is no storage quota on log entries that would cause time travel to fail. Option D is incorrect because Delta time travel is time-based (controlled by retention duration), not snapshot-count-limited; there is no fixed snapshot cap of 30.

55 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free