Data Processing: free practice questions

AQE's skew join optimization specifically targets data skew in shuffle-based joins. It detects which partitions are significantly larger than average, splits those large partitions into smaller chunks, and duplicates the matching partition from the other side so that work can be parallelized — directly solving the straggler task problem. Option A describes dynamic partition coalescing, which merges many small partitions into fewer larger ones — the opposite problem (too many small tasks, not a few huge ones). Option C is incorrect because AQE can convert joins to broadcast joins based on runtime size statistics, but this applies to cases where one side is small enough to broadcast; it does not help when one side has skewed but large partitions. Option D describes predicate pushdown, which is a Catalyst optimizer feature, not an AQE runtime skew optimization.

Exactly-once semantics and state recovery in Structured Streaming depend entirely on a configured checkpoint location (`checkpointLocation`). Without it, there is no WAL (write-ahead log) or state store, so on restart the job has no memory of prior progress or aggregation state and begins from scratch. Option A is wrong: `Trigger.Once()` does support stateful operations and respects checkpoints. Option C is wrong: `complete` output mode does rewrite the full result set each trigger, but this is unrelated to state being lost on restart — checkpointing still preserves state across restarts in complete mode. Option D is wrong: Delta Lake sources are fully supported for stateful Structured Streaming jobs.

In Structured Streaming stream-stream joins, state grows unboundedly unless Spark knows when it is safe to evict unmatched records from both sides. This requires: (1) `withWatermark` applied to the event-time column on BOTH input streams, AND (2) a time-range join condition (inequality predicate on the time columns) that bounds how far apart matching events can be. Together, these allow Spark to determine that once the watermark has advanced sufficiently, any unmatched record cannot receive a future match and can be dropped from state. Option A is wrong: `dropDuplicates` removes duplicate keys within a single stream but does not bound join state, which accumulates all unmatched records waiting for a partner. Option C is wrong: `spark.sql.streaming.stateStore.maxKeys` is not a valid Spark parameter; there is no built-in key cap for state stores. Option D is wrong: trigger intervals control micro-batch cadence, not state eviction; state accumulates across triggers regardless of the trigger interval.

delta.logRetentionDuration controls how long the Delta transaction log (and therefore time-travel history) is preserved — setting it to 60 days allows queries against snapshots up to 60 days old. delta.deletedFileRetentionDuration controls how long data files that are no longer referenced by the current table version are kept before VACUUM can remove them — setting it to 7 days means VACUUM can clean up stale files older than 7 days. Option B reverses the two properties, which would restrict time travel to only 7 days and retain deleted file data for 60 days — the opposite of the requirement. Option C uses invented property names (delta.dataRetentionDuration and delta.vacuumRetentionDuration) that do not exist in Delta Lake. Option D only sets log retention; without also adjusting deletedFileRetentionDuration, VACUUM defaults to retaining files for 7 days anyway, but it does not give the team explicit, declarative control over both settings together, and manually scheduling VACUUM with RETAIN 168 HOURS overrides the table property per-run rather than storing it persistently.

The key performance improvement of Pandas UDFs is the use of Apache Arrow for JVM-to-Python data transfer. Arrow's columnar, zero-copy-friendly format allows entire batches of rows to be transferred at once rather than serializing/deserializing each row individually (as standard Python UDFs do via Pickle). Once data is in Python, Pandas and NumPy operations are vectorized, further improving throughput. Option B is wrong because Pandas UDFs do NOT bypass the Python interpreter — they still execute in a Python worker process; the improvement comes from batch transfer and vectorized operations, not JVM compilation. Option C is wrong because standard Python UDFs do NOT run on the JVM — they are deserialized row-by-row into a Python worker process using Pickle serialization, which is exactly what makes them slow. Tungsten/code generation applies to JVM-native operations, not Python UDFs. Option D is wrong because Python UDFs (both standard and Pandas) run in a separate Python worker process, not in the JVM memory space; they do NOT share JVM memory. Arrow reduces serialization cost but still involves inter-process communication.