Databricks Data Engineer Associate: Working with Lakeflow Jobs Practice Questions

Databricks Workflows provides the Task Values API (dbutils.jobs.taskValues.set() and dbutils.jobs.taskValues.get()) specifically for passing lightweight key-value outputs between tasks in the same job run. Writing to a Delta table works functionally but is not the intended mechanism for inter-task communication and introduces unnecessary overhead. Secret scopes are for storing credentials, not dynamic run-time values produced by tasks. Job parameters are set at the job level before the run begins and cannot be dynamically set by one task for consumption by another during the same run.

Setting 'Run as' to a service principal ensures all job actions use that identity instead of the job creator's credentials, satisfying rule 1. Attaching the IAM instance profile to the job cluster definition gives the cluster nodes the correct AWS permissions to access S3, satisfying rule 2. Configuring a job-level timeout of 240 minutes (4 hours) causes Databricks to automatically terminate the run if it exceeds that duration, satisfying rule 3. Adding a monitoring task is a custom workaround that is unnecessary because native job-level timeout exists. Unity Catalog enforces data governance but does not automatically change the run-as identity for a job — that must be explicitly configured. spark.databricks.timeout is not a valid Spark configuration property for controlling job run duration in Databricks Workflows.

Setting maximum concurrent runs to 1 ensures only one instance of the job can execute at a time; if a scheduled trigger fires while a run is active, the new run is queued rather than started immediately. There is no native 'Skip if already running' checkbox in Databricks Workflows — this is a distractor conflating behavior from other orchestrators. Autoscaling compute does not control concurrency at the job scheduling level. A retry policy with zero retries and a long timeout only affects failure handling, not concurrent execution.

Job-level 'On failure' email notifications ensure the on-call engineer is alerted whenever any task in the workflow fails, satisfying requirement 1. Setting non-critical tasks to 'Continue on failure' (available in task settings) allows downstream tasks to still execute even if the non-critical task fails, satisfying requirement 2. Setting 'Run if' to 'At least one parent succeeded' controls downstream execution logic based on parent outcomes but is designed for branching logic, not for allowing a failed task itself to not block — 'Continue on failure' is the direct setting. Configuring retries on all tasks would reduce failure notifications but adds unnecessary delay and does not address the notification or unblocking requirements. Using a Databricks SQL Alert is for query-based threshold monitoring, not Workflow task failure alerting.

A Full refresh in DLT causes the pipeline to drop and recompute all tables from source data, effectively re-processing everything from scratch. Switching to Continuous mode changes the execution frequency but still processes data incrementally using checkpoints — it does not force a full recompute. Manually deleting streaming checkpoints is not a supported or recommended practice in DLT; DLT manages its own internal state. Changing the target schema creates a new set of tables but does not force re-processing of the existing pipeline's data within the same pipeline definition.