Cost and Performance Optimization – DE Professional Study Guide
This domain accounts for 13% of the exam — the second-largest — and covers making pipelines fast and cheap through Delta layout, cluster choices, and Spark tuning.
Delta Layout Optimization
- OPTIMIZE — compacts many small files into fewer larger ones (fixes the small files problem from streaming micro-batches). Optimized writes / auto-compaction prevent them.
- ZORDER BY — co-locates data on frequently filtered (high-cardinality) columns to improve data skipping. (Liquid clustering is the newer alternative.)
- Data skipping relies on per-file min/max statistics in the transaction log to skip files that can't match a filter.
- VACUUM removes old, unreferenced data files (after the retention window) — controls storage cost (and completes PII purges).
Unlock with a paid plan
3 more sections (about 131 words) continue below. Every paid plan unlocks all Data Engineer Professional study guides, the full question bank and mock exams, from $4.97/month.