Deploy and maintain data pipelines and workloads Study Guide — DP-750

Deploy and Maintain Data Pipelines and Workloads

This domain accounts for 33% of the DP-750 exam — the largest single domain. It tests your ability to design, implement, monitor, troubleshoot, and optimize end-to-end data pipelines on the Databricks Lakehouse Platform using Lakeflow Jobs, Declarative Pipelines (DLT), Databricks Asset Bundles, and supporting Azure services. Expect scenario-based questions that require you to choose the right tool, configuration, or troubleshooting step given a specific constraint such as cost, latency, reliability, or governance.

Pipeline Order of Operations: Notebooks vs. Declarative Pipelines

Core Concepts

A Databricks notebook is imperative: cells execute in explicit sequence, and the developer controls ordering, error handling, and retries. A Declarative Pipeline (built with Delta Live Tables / Lakeflow Declarative Pipelines) is declarative: you define what data transformations should produce, and the runtime computes the dependency graph and execution order automatically.

Pipeline order of operations (DLT):

1. Parse pipeline definition and resolve dataset references.
2. Build the lineage DAG.
3. Execute sources (bronze) → intermediate (silver) → final (gold) in topological order.
4. Apply data quality expectations during each dataset materialization.
5. Commit to Delta, update event log, report metrics.

Common Traps

• Assuming notebooks automatically handle retries or schema drift — they do not without explicit code.
• Confusing LIVE. prefix (required for DLT datasets) with standard Delta table references.
• Believing DLT can replace a notebook for non-tabular tasks (e.g., REST API calls, file renaming) — it cannot; use notebooks or Python tasks in Jobs for those steps.
• Expecting DLT pipelines to run ad-hoc queries interactively — pipelines run in isolated pipeline clusters.

Worked Scenario

Requirement: A team needs to ingest raw JSON from Azure Data Lake, apply data quality rules that drop bad records, and produce a curated Delta table, with automatic schema evolution. Reasoning: Declarative Pipeline with STREAMING TABLE backed by Auto Loader handles schema evolution (cloudFiles.schemaEvolutionMode = addNewColumns), and DLT expectations with ON VIOLATION DROP ROW removes bad records without custom code. A plain notebook would require manual checkpoint management and custom schema handling.

Lakeflow Jobs: Task Logic, Triggers, Schedules, Alerts, Automatic Restarts

Core Concepts

A Lakeflow Job (formerly Databricks Workflows Job) is the orchestration unit. A job contains one or more tasks connected by dependency edges. Each task runs a specific compute action.

Supported task types: Notebook, Python script, Python wheel, Spark JAR, SQL, Delta Live Tables pipeline, dbt, Databricks SQL alert, Run Job (nested), If/Else condition, For Each loop, Dashboard refresh.

Trigger types:

Alerts and notifications: Configured per-job or per-task. Supported channels: email, Slack (webhook), Microsoft Teams (webhook), PagerDuty. Alert events: run start, success, failure, duration exceeded, streaming backlog exceeded.

Automatic restarts: Set max_retries on a task (0 = no retry). Set retry_on_timeout: true to retry timed-out runs. min_retry_interval_millis controls the backoff. At job level, max_concurrent_runs controls parallelism (default 1).

Common Traps

• Confusing continuous trigger (job-level) with a streaming task inside a job — both can handle streaming but serve different purposes.
• Setting max_retries at the job level expecting per-task retry — retries are per task, not per job.
• Believing file-arrival trigger is a pure streaming trigger — it polls storage; latency is seconds to minutes, not milliseconds.
• Forgetting that nested jobs (Run Job task type) propagate failure to the parent by default.