Data Engineer Professional · 10% of the exam

Monitoring and Logging: free practice questions

5 sample questions from our 25-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Databricks exam.

1. A data engineer is using a StreamingQueryListener to monitor a Structured Streaming job. In the `onQueryProgress` callback, they observe that `inputRowsPerSecond` is consistently near zero while `processedRowsPerSecond` is also near zero, but the `batchDuration` metric is very high (over 45 seconds per batch). The source is a Kafka topic that is confirmed to have active producers. What does this pattern MOST LIKELY indicate?

  • A. The streaming job has no active trigger configured and is defaulting to once-per-day processing
  • B. The Kafka consumer group is being rebalanced repeatedly, causing the streaming source to pause consumption
  • C. The bottleneck is in the stateful transformation or sink write phase—the job is spending most of each batch duration processing or writing, not reading from Kafka✓ Correct
  • D. The StreamingQueryListener is capturing stale metrics from a previous query run that was not properly stopped
Explanation

When `inputRowsPerSecond` is near zero but `batchDuration` is very high, the Kafka source itself is not the bottleneck—instead, each batch is taking a long time to complete even with minimal new input. This indicates the processing logic (e.g., a complex stateful aggregation, large state store checkpoint, or slow sink write such as a Delta MERGE) is consuming most of the batch time. Option A is incorrect; Structured Streaming defaults to a continuous micro-batch trigger, not once-per-day. Option B (Kafka rebalance) would typically manifest as failed batches, exceptions in the query progress, or an `inputRowsPerSecond` of exactly zero with source lag growing—not a consistently high `batchDuration`. Option D is implausible because a stopped query would not continue emitting `onQueryProgress` events.

2. A data engineer reviews the cluster event logs in the Databricks UI for a job cluster and notices a sequence of events: `RESIZING` → `RUNNING` → `RESIZING` → `RUNNING` repeated multiple times during the job. The job is configured with autoscaling between 2 and 20 workers. What does this event pattern MOST LIKELY indicate?

  • A. The cluster hit the maximum worker limit (20) and Databricks automatically restarted the cluster with a larger instance type
  • B. The autoscaler is repeatedly scaling up and then scaling back down within the same job run, which may indicate bursty resource demand or a misconfigured scale-down delay✓ Correct
  • C. The Spark driver is crashing and restarting, causing executor reassignment events to appear as RESIZING
  • D. The job cluster is shared with another concurrent job, leading to resource contention and forced resizing events
Explanation

Repeated `RESIZING` → `RUNNING` transitions in a cluster event log during a single job run is the expected signature of autoscaler activity—scaling up when tasks queue up and scaling down when resources are idle. If this occurs many times, it often means the workload has bursty parallelism patterns, or the `spark.databricks.aggressiveWindowDownS` (scale-down delay) is set too low, causing premature scale-down between bursts. Option A is incorrect; hitting the max worker limit does not cause a cluster restart or instance type change. Option C is wrong; driver crashes generate `DID_NOT_START` or `DRIVER_HEALTHY`/`DRIVER_UNHEALTHY` events, not `RESIZING`. Option D is wrong; job clusters are single-tenant and are not shared across concurrent jobs (unlike all-purpose clusters).

3. A streaming job using Structured Streaming writes to a Delta Lake table. The on-call engineer wants to programmatically capture metrics such as input rows per second, processing time, and watermark lag for every micro-batch without modifying the dashboard. Which Spark API should be used?

  • A. Implement a StreamingQueryListener and override onQueryProgress() to extract metrics from the StreamingQueryProgress object, then publish them to an external monitoring system.✓ Correct
  • B. Enable spark.sql.streaming.metricsEnabled=true and scrape the metrics from the Ganglia UI on the cluster.
  • C. Call streamingQuery.recentProgress on a fixed polling interval inside a separate thread and log the JSON output.
  • D. Attach a foreachBatch function to the stream and query spark.sparkContext.statusTracker() inside each batch to retrieve streaming metrics.
Explanation

StreamingQueryListener (option A) is the idiomatic, event-driven Spark API for monitoring Structured Streaming. Registering a listener and overriding onQueryProgress() gives access to the full StreamingQueryProgress object — containing inputRowsPerSecond, processingTime, eventTime watermark, and more — on every micro-batch, with no polling required and no dashboard changes. Option B (Ganglia + metricsEnabled) does expose some streaming metrics via the metrics system, but Ganglia is not available on Databricks clusters by default (it was deprecated), and scraping it is not a programmatic API approach. Option C (recentProgress polling) is a valid quick-and-dirty technique but is a polling approach with potential race conditions and missed batches, making it less reliable than a listener. Option D is incorrect because spark.sparkContext.statusTracker() returns Spark job/stage status for the driver, not Structured Streaming query metrics; also, using foreachBatch for monitoring adds overhead to the streaming write path.

4. A data engineer is reviewing the Spark UI for a job that performs a large shuffle join between two Delta tables. The Stages tab shows that Stage 4 has a median task duration of 8 seconds, but the 75th-percentile task duration is 4 minutes and 12 seconds. The DAG shows Stage 4 immediately follows a ShuffleMapStage. What is the MOST LIKELY root cause, and what should the engineer investigate first?

  • A. Insufficient driver memory causing spill to local disk on the driver node
  • B. Data skew in the shuffle partition keys, causing a small number of tasks to process disproportionately large amounts of data✓ Correct
  • C. Too many executor cores per node, leading to CPU contention during shuffle read
  • D. The shuffle service is unavailable, forcing tasks to re-fetch blocks from remote executors
Explanation

The large gap between median task duration (8 s) and the 75th-percentile (4+ min) within a single stage following a shuffle is the classic signature of data skew. A small number of shuffle partitions receive a far larger share of the data due to non-uniform key distribution, making those tasks take much longer than the rest. Option A is incorrect because driver memory issues manifest as driver OOM errors or slow job submission, not long individual task durations. Option C (too many cores per executor) would cause uniformly slower tasks across the board, not a bimodal distribution. Option D (shuffle service unavailable) would typically cause task failures or retries, not a skewed duration distribution.

5. A data engineering team runs 50 Delta Live Tables pipelines across multiple business units. The finance team needs a weekly report showing the DBU consumption broken down by individual pipeline. Which approach is MOST appropriate for obtaining this data?

  • A. Query the Databricks Billing API (or the system.billing.usage table if available) filtering on usage_metadata.dlt_pipeline_id to aggregate DBUs per pipeline per week.✓ Correct
  • B. Sum the number of cluster events in each pipeline's cluster event log and multiply by a fixed DBU rate to estimate consumption.
  • C. Use the DLT event log and sum the processing_time field across all flow_progress events to derive a proxy for DBU usage.
  • D. Navigate to the Databricks Workflows UI for each pipeline and manually read the cluster runtime from each run's summary page.
Explanation

The system.billing.usage table (or the Databricks Billing API) contains per-workload DBU consumption records with metadata including pipeline identifiers (usage_metadata.dlt_pipeline_id), making it the correct and scalable source for pipeline-level cost attribution (option A). Option B is incorrect because cluster events are lifecycle records (e.g., STARTED, RESIZED) and not time-series utilization metrics; multiplying event counts by a DBU rate is not a valid computation. Option C is incorrect because the DLT event log's processing_time tracks execution duration of individual flow runs, not DBU consumption — the conversion would require cluster size and type information not present in the event log. Option D is operationally impractical for 50 pipelines and produces only rough wall-clock time, not DBU figures.

20 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free