Model Development Study Guide — Machine Learning Professional

Model Development – Databricks ML Professional Study Guide

This domain accounts for 44% of the exam (tied for the largest) and covers building ML at scale on Databricks: Spark ML pipelines, distributed training and hyperparameter tuning, advanced MLflow tracking, and the Feature Store. This is the Professional exam, so questions go beyond "what is X" to "how do you do X correctly at scale and reproducibly." Master MLflow's advanced features and the Feature Store and you cover most of this domain.

Spark ML Pipelines

• Transformer — implements transform() to produce a new DataFrame (feature steps, or a fitted model producing predictions).
• Estimator — implements fit() to learn from data and return a Transformer (the trained model).
• Pipeline — chains Transformers and an Estimator into stages; fitting yields a PipelineModel that applies the identical sequence at inference. Putting encoders/assemblers inside the Pipeline is how you prevent training/serving skew.

Distributed Training and Tuning

• Distributed training — for deep learning, use framework integrations (e.g., TorchDistributor, Horovod) to parallelize across cluster workers rather than training on the driver alone.
• Hyperparameter tuning at scale — Hyperopt with SparkTrials distributes tuning trials across the cluster in parallel, integrated with MLflow. The objective returns a loss (lower is better) that Hyperopt minimizes — negate metrics like accuracy.