Data Engineer Professional · 10% of the exam

Data Transformation, Cleansing, and Quality: free practice questions

5 sample questions from our 81-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Databricks exam.

1. A data engineer runs `df.persist(StorageLevel.DISK_ONLY)` on a 50 GB DataFrame before two downstream transformations. A colleague suggests switching to `df.cache()` instead. What is the PRIMARY difference between these two calls in a default Databricks cluster configuration?

  • A. `cache()` stores data in memory only (MEMORY_AND_DISK is not available on Databricks), while `persist(DISK_ONLY)` stores data only on disk.
  • B. `cache()` is equivalent to `persist(MEMORY_AND_DISK)` with deserialized storage, and will spill to disk only when memory is insufficient, whereas `persist(DISK_ONLY)` never uses memory — data is always read from disk for each downstream action.✓ Correct
  • C. `persist(DISK_ONLY)` automatically replicates the data across two executor nodes for fault tolerance, while `cache()` does not.
  • D. `cache()` persists the DataFrame across Spark sessions, while `persist(DISK_ONLY)` is scoped to the current session only.
Explanation

`df.cache()` in Spark is shorthand for `persist(StorageLevel.MEMORY_AND_DISK)`, which stores deserialized partitions in JVM heap memory and spills to executor local disk only when memory pressure occurs. `persist(DISK_ONLY)` always writes serialized partitions to disk and reads them back from disk for every downstream use, which avoids memory pressure but incurs higher I/O cost. For a 50 GB DataFrame on a cluster with sufficient memory, `cache()` would be faster; if memory is insufficient, both ultimately use disk but `cache()` still tries memory first. Option A is incorrect: MEMORY_AND_DISK is the default for `cache()` on Databricks. Option C is incorrect: DISK_ONLY_2 provides replication, not plain DISK_ONLY. Option D is incorrect: neither call persists data across sessions; both are scoped to the current SparkContext.

2. Which DLT expectation action would cause the pipeline update to FAIL when records violate a critical quality rule?

  • A. The fail action (EXPECT ... ON VIOLATION FAIL UPDATE)✓ Correct
  • B. The drop action
  • C. The allow (warn) action
  • D. No action is possible
Explanation

The fail action stops the pipeline update when a critical expectation is violated, preventing bad data from propagating. Drop (B) removes violating rows but continues, the default/warn action (C) only records metrics, and DLT does support failing (D is false).

3. Which of the following is a correct description of how Adaptive Query Execution (AQE) handles skewed joins in Spark?

  • A. AQE detects skewed partitions at query planning time by analyzing table statistics and pre-splits them before the shuffle occurs.
  • B. AQE detects skewed partitions at runtime after the shuffle map stage completes, then automatically splits the skewed partitions into smaller sub-tasks and replicates the corresponding build-side data to process them in parallel.✓ Correct
  • C. AQE eliminates skew by converting every sort-merge join into a broadcast hash join when skew is detected.
  • D. AQE handles skew by repartitioning the entire DataFrame using a salting key automatically chosen by the Catalyst optimizer.
Explanation

AQE's skew join optimization works at runtime: after shuffle map tasks complete, the Spark runtime inspects the actual partition sizes. Partitions exceeding a skew threshold (controlled by `spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes` and `skewedPartitionFactor`) are split into smaller sub-tasks, and the corresponding partition from the other side of the join is replicated for each sub-task. This happens without any code changes. Option A is wrong: AQE is specifically a runtime optimization; static query planning uses cost-based optimization (CBO) with statistics but cannot split partitions after the fact. Option C is wrong: AQE can convert sort-merge joins to broadcast joins (a separate AQE feature), but skew handling is distinct and does not require this conversion. Option D is wrong: salting is a manual developer technique; AQE does not automatically introduce salt keys.

4. A data engineer is building a production pipeline that uses Structured Streaming with checkpointing. After a code change that alters the schema of an intermediate aggregation, the engineer restarts the streaming job pointing to the same checkpoint directory. What is the most likely outcome?

  • A. The job resumes seamlessly from the last committed offset because checkpointing only stores offset information, not schema or state.
  • B. The job fails to start or throws an AnalysisException because the new query's schema or logical plan is incompatible with the state stored in the checkpoint, which encodes the previous query's plan and state schema.✓ Correct
  • C. The job silently discards the old checkpoint and starts reprocessing all data from the beginning of the source.
  • D. The job resumes from the checkpoint but automatically migrates the state schema to match the new query.
Explanation

Spark Structured Streaming checkpoints store not only source offsets but also the logical plan (query identity) and the schema of stateful operator state (e.g., aggregation state). When a breaking change is made to the query — such as modifying the schema of an aggregation — the checkpoint becomes incompatible and the job will fail with an AnalysisException or StreamingQueryException on restart. The checkpoint directory must be deleted and data reprocessed (or a schema migration procedure followed for minor compatible changes). Option A is wrong because checkpoints store far more than just offsets — they include the query plan and state data. Option C is wrong because Spark does NOT silently discard incompatible checkpoints; it raises an error. Option D is wrong because Spark does not automatically migrate state schemas; this is the engineer's responsibility.

5. A data engineer is comparing the performance of a standard Python UDF and a Pandas UDF (vectorized UDF) for a transformation applied to 500 million rows in a Spark DataFrame. Which statement BEST explains why the Pandas UDF typically outperforms the standard Python UDF for this workload?

  • A. Pandas UDFs execute natively on the JVM using the Catalyst optimizer, while standard Python UDFs are interpreted by CPython.
  • B. Pandas UDFs transfer data between the JVM and Python in columnar Arrow batches, reducing serialization overhead compared to standard Python UDFs which serialize one row at a time.✓ Correct
  • C. Pandas UDFs bypass the Python process entirely and run directly in the Spark executor's JVM thread pool.
  • D. Pandas UDFs are automatically compiled to native machine code by the Photon engine, whereas standard UDFs are not.
Explanation

Standard Python UDFs serialize each row individually between the JVM (Spark executor) and the Python worker process, resulting in enormous serialization overhead at scale. Pandas UDFs (vectorized UDFs) use Apache Arrow to transfer data in columnar batches between the JVM and Python, dramatically reducing per-row serialization cost and enabling vectorized Pandas operations on entire batches. Option A is wrong — both UDF types run in Python (CPython), not on the JVM; neither uses Catalyst directly. Option C is wrong — Pandas UDFs still execute in a separate Python process (via Arrow-based IPC); they do not run in the JVM. Option D is wrong — Photon does not compile Python UDFs to native code; Photon accelerates certain Spark SQL operations that avoid UDFs entirely.

76 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free
Data Transformation, Cleansing, and Quality — Free Data Engineer Professional Practice Questions | DataCertPrep — Certification Prep