Databricks Data Engineer Professional: Monitoring and Alerting Practice Questions

pipeline_failure and error are the two event types most directly associated with a pipeline entering ERROR state. pipeline_failure is emitted when the pipeline terminates abnormally, and error events contain detailed exception messages and stack traces for individual failures within the pipeline. flow_progress events track the execution state of individual flows (e.g., RUNNING, COMPLETED) but do not themselves contain failure details. dataset_definition events record schema and dataset metadata at pipeline startup and are unrelated to failures. cluster_info events record cluster configuration and are useful for infrastructure debugging but not for identifying pipeline logic errors. user_action events record manual user interventions (e.g., start/stop) and are not indicative of root cause.

High GC time on a single executor (while peers are healthy) almost always points to that executor handling more data than others, either due to data skew or due to being pinned to large cached/broadcast data. (D) Increasing shuffle partitions reduces the data volume per partition, which can reduce per-executor heap pressure if skew is involved. (E) Checking whether the executor holds oversized broadcast or cached data identifies a common root cause. (A) Reviewing task metrics for data volume imbalance directly confirms or rules out skew. Option C (spark.executor.memoryOverhead) allocates off-heap/container memory for Python processes and native memory—it does not increase JVM heap and would not help GC pressure in the JVM. Option F (reducing cores) would lower throughput without addressing the root cause. Option B (replacing the instance type) may be a last resort but is premature before diagnosis; also, since only one executor is affected with identical hardware, it is not a hardware limitation.

The Query Profile (option C) in Databricks SQL provides a visual physical plan with node-level metrics, making it straightforward to identify regressions like new full scans, missing file pruning, or skew — all common causes of sudden slowdowns. Query History duration and rows-read metrics (option B) let engineers compare the current execution against historical baselines to confirm and quantify the regression. Cluster event logs (option E) are useful for infrastructure-level events but are not exposed directly in Databricks SQL Query History; also, SQL warehouse drivers rarely throw OOM errors from a single query slowdown. DESCRIBE HISTORY (option D) may reveal file-layout changes but is not part of Query History tooling and is a secondary investigation step, not a first-line diagnostic inside the SQL interface. The billing API (option A) measures DBU cost, not query-level performance metrics, and cannot pinpoint the cause of a slow query.

DESCRIBE HISTORY is the correct command. It returns a full audit trail of all operations on a Delta table—including INSERT, UPDATE, DELETE, MERGE, OPTIMIZE, schema changes, and more—along with timestamps, user, operation parameters, and metrics. Option D is incorrect because transactions.event_log is not a valid Delta Lake construct; event logs apply to DLT pipelines, not standalone Delta tables. Option C (SHOW TBLPROPERTIES) only displays key-value table properties, not operation history. Option B reads raw JSON transaction log files directly, which is possible but not reliable or complete for a 30-day audit window, and is not a supported or recommended approach.

High GC time on an executor indicates JVM heap pressure. The correct mitigations are: (F) Increasing executor memory gives the JVM more headroom before GC kicks in, directly reducing GC frequency. (C) Tuning the GC algorithm (e.g., switching to ZGC) can reduce pause times and improve throughput, which is a standard JVM performance technique. (A) Reducing object churn — by caching DataFrames to avoid recomputation or using off-heap storage to keep data outside the JVM heap — directly reduces the number of objects the GC must manage. Option D is incorrect: increasing executor cores does not help with GC; GC threads are managed by the JVM independently, and adding task threads would actually compete for CPU with GC, potentially worsening latency. Option B is incorrect: killing and restarting an executor temporarily clears the heap but does not address the root cause; the GC problem will recur immediately. Option E is partially logical but misguided — reducing partitions would make each partition larger, not smaller, potentially increasing per-task object creation; the correct direction would be to increase partitions to reduce per-partition data.