Set up and configure an Azure Databricks environment Study Guide — DP-750

Set Up and Configure an Azure Databricks Environment

This domain tests your ability to select appropriate compute types, tune performance settings, manage libraries and access, and design Unity Catalog hierarchies. It accounts for 17% of the exam score, making it the second-largest domain. Questions appear as scenario-based multiple-choice items that ask you to choose the correct compute type, catalog object type, or DDL statement for a given business requirement. Expect one or two questions on exact defaults and limits.

Compute Types: Job, Serverless, Warehouse, Classic, Shared

Azure Databricks exposes several distinct compute surfaces. Selecting the wrong one is the most common mistake on the exam.

Key Distinctions to Memorize

• Serverless compute for notebooks/jobs spins up in seconds because infrastructure is pre-provisioned by Databricks. The customer does not see or manage the underlying VMs.
• SQL Warehouse types: Classic (Databricks-managed VMs, slower start), Pro (adds Query Federation and larger result caching), Serverless (fastest start, no infrastructure). The exam distinguishes Pro from Serverless warehouses.
• Shared access mode on a classic cluster uses process isolation per user; it supports Python and SQL but not Scala or R in user code (Scala is reserved for the driver process).
• Single-user access mode supports all languages including Scala and R but binds the cluster to one user principal.
• No-isolation shared (legacy) is deprecated for Unity Catalog environments.

Decision flow for selecting compute:

Use this flowchart to decide which compute surface to provision.

Common Traps

• The exam may describe "a cluster shared by a team" and offer both Shared access mode and No-isolation shared. Always choose Shared access mode; no-isolation is deprecated.
• "Serverless" appears for both notebook/job compute AND SQL warehouses — they are separate features. Read the question carefully to know which surface is meant.
• Job compute cannot be started or attached interactively. If a question says "engineers open notebooks and run ad hoc queries," job compute is wrong.

Compute Performance Settings: Autoscaling, Termination, Node Types, Pooling

Autoscaling

Standard autoscaling scales between a configured minimum and maximum worker count. Databricks uses cluster metrics and pending task queues to trigger scale-up; scale-down requires idle workers.

Optimized autoscaling (available on job compute and in Premium tier) scales down more aggressively: it removes workers after they are idle for as little as 40 seconds instead of waiting for the full scale-down interval.

Auto-Termination

• Default auto-termination: 120 minutes for all-purpose clusters.
• Setting termination to 0 disables auto-termination (runs indefinitely).
• Job clusters automatically terminate after the job completes regardless of this setting.
• Serverless compute has no termination setting — it scales to zero automatically.

Node Types and Memory/CPU Selection

• Driver node coordinates task distribution; it should be memory-optimized when the driver aggregates large results.
• Worker nodes execute tasks; choose memory-optimized for shuffle-heavy jobs, compute-optimized for CPU-intensive ML.
• Spot instances (Azure Spot VMs) reduce cost; use with job clusters where interruption is tolerable. All-purpose clusters with spot workers may lose work on interruption.

Instance Pools

An instance pool maintains a set of idle, ready-to-use VM instances that clusters can acquire to reduce start-up time.

• Pool instances are billed at the DBU rate for the pool tier when idle (not at cluster DBU rates).
• Clusters using a pool inherit the VM type and optionally the Databricks Runtime from the pool.
• A pool can have a minimum idle instance count (always warm) and a maximum capacity.
• Pools do not eliminate cold starts entirely — they eliminate the VM provisioning delay but Databricks Runtime still initializes.

Common Traps

• Auto-termination applies to all-purpose clusters, not job clusters. Exam distractors will ask about terminating a job cluster — the answer is it terminates on job completion, not via the termination timer.
• Pools reduce start time but do not guarantee zero start time.
• Optimized autoscaling is only available in Premium tier. Standard tier uses standard autoscaling.

Worked scenario: A data engineering team runs nightly ETL jobs that take 15–45 minutes with highly variable data volumes. The priority is cost efficiency. → Choose job compute with optimized autoscaling and spot worker instances. Optimized autoscaling removes idle workers quickly; spot instances cut VM cost; job clusters terminate on completion.