1. A data engineer is designing a Delta Lake table for an event logging system. The team inserts millions of small individual event records throughout the day using streaming micro-batches, resulting in thousands of tiny Parquet files accumulating over several weeks. Which of the following approaches BEST addresses this small-file problem while maintaining Delta Lake capabilities? (Select TWO)
- A. Run the OPTIMIZE command on the table periodically (e.g., nightly), which rewrites small files into larger, right-sized Parquet files.✓ Correct
- B. Convert the Delta table to CSV format, which inherently avoids small-file overhead by appending to a single file.
- C. Enable Auto Optimize (Auto Compaction and Optimized Writes) on the table so that Delta Lake automatically compacts small files during and after write operations.✓ Correct
- D. Run VACUUM with a retention of 0 hours to immediately delete all small files and force Delta Lake to regenerate consolidated files.
- E. Increase the number of streaming micro-batch partitions to 10,000 to distribute the small files more evenly across worker nodes.
Explanation
Correct: (A) The OPTIMIZE command explicitly compacts small Parquet files into larger target-size files (default ~1 GB), directly solving the small-file problem. It can be scheduled to run periodically without disrupting reads. (C) Auto Optimize combines Optimized Writes (which reduces the number of files written per operation) and Auto Compaction (which compacts files after writes), proactively preventing and reducing small-file accumulation without manual scheduling. Wrong: (B) Converting to CSV would lose all Delta Lake features (ACID, time travel, schema enforcement) and CSV does not append to a single file—it creates many files just like Delta. (D) Running VACUUM with 0-hour retention deletes old data files but does NOT consolidate or compact existing small files; it would also bypass the default 7-day safety threshold and could break time travel. (E) Increasing the number of partitions would create MORE small files, not fewer—this worsens the problem.