DP-750 Prepare and process data Study Guide

Prepare and Process Data

This domain covers the full data pipeline lifecycle inside the Databricks Lakehouse: how data enters the platform, how it is stored and organized, how it is transformed and cleansed, and how quality is enforced. It represents 33% of the DP-750 exam, making it the single largest domain. Expect questions that require you to choose the correct ingestion tool for a scenario, design partition and clustering strategies, apply SCD logic, configure streaming pipelines, and enforce schema and data quality rules in Unity Catalog.

Ingestion Design: Extraction Types, File Types, and Tool Selection

Extraction Types

Type	Description	Typical Use
Full extraction	Entire source copied each run	Small tables, no change tracking available
Incremental (delta) extraction	Only changed records since last run	Large tables with timestamps or sequence keys
CDC (Change Data Capture)	Row-level insert/update/delete from source log	High-fidelity replication, low latency
Event-based	Messages published to a queue/stream	Real-time or near-real-time pipelines

File Format Decision Matrix

Format	Schema enforcement	Splittable	ACID	Best for
Delta	Yes (schema-on-write)	Yes	Yes	Lakehouse tables, default choice
Parquet	Yes (schema-on-read)	Yes	No	Interop, read-heavy analytics
Iceberg	Yes	Yes	Yes	Multi-engine open table format
CSV	No	Yes (by line)	No	Landing/raw zone only
JSON	No (semi-structured)	No (single record)	No	APIs, nested data landing

Tool Selection: Lakeflow Connect vs Notebooks vs ADF

Tool	Best fit	Limitations
Lakeflow Connect	Managed SaaS connectors (Salesforce, ServiceNow, databases) — no-code ingestion into Unity Catalog	Limited to supported connector catalog; no custom transformation logic
Notebooks / Spark	Custom logic, complex transformations, any source reachable by driver	Requires engineering effort; not a managed connector service
Azure Data Factory (ADF)	Orchestrating non-Databricks systems, copying from Azure-native sources, existing ADF investments	Not native to Databricks; cannot write Delta natively without linked service config
Lakeflow Declarative Pipelines	Streaming or batch pipelines with built-in quality checks and Auto Loader	Databricks-only; not appropriate for purely external orchestration

Common traps

Exam distractors often offer ADF when the scenario specifies Unity Catalog-native lineage — prefer Lakeflow Connect or Pipelines.
Lakeflow Connect is a managed connector service, not a transformation engine. Do not select it for questions asking about custom business logic.
CSV is never a good landing-zone target format; it is a source format.

Batch vs Streaming Loading and Table Formats

Batch vs Streaming Comparison

Dimension	Batch	Structured Streaming
Latency	Minutes to hours	Seconds to sub-second
Trigger	Schedule / manual	`Trigger.Once`, `Trigger.AvailableNow`, continuous
State management	None required	Stateful (watermarks, checkpoints)
Fault tolerance	Re-run job	Checkpoint-based exactly-once
Typical sources	Files, JDBC	Kafka, Event Hubs, Auto Loader

Delta Lake Key Behaviors

ACID transactions: every write is atomic; concurrent readers always see a consistent snapshot.
Time travel: VERSION AS OF n or TIMESTAMP AS OF '2024-01-01' — default 30-day retention (configurable via delta.logRetentionDuration).
Transaction log: stored in _delta_log/; every commit is a JSON file; checkpointed every 10 commits by default.
Schema enforcement: by default Delta rejects writes whose schema does not match — no silent data corruption.
Schema evolution: enabled with .option("mergeSchema", "true") or TBLPROPERTIES ('delta.schema.autoMerge.enabled' = 'true').

Event Hubs with Structured Streaming

Use the Kafka-compatible endpoint of Azure Event Hubs with spark.readStream.format("kafka"). The consumer group, topic (= Event Hub name), and bootstrap server (= <namespace>.servicebus.windows.net:9093) must be configured. Checkpointing is mandatory for exactly-once semantics.