Question 1 · Model Development
A Spark ML pipeline must chain a StringIndexer, a VectorAssembler, and a classifier so the same transformations apply consistently at train and inference time. Which construct enforces this?
Show answer & explanation
Correct answer: B
- B. A Spark ML Pipeline (stages of Transformers and an Estimator)
A Spark ML Pipeline bundles Transformers and an Estimator into stages. Fitting it produces a PipelineModel that applies the same sequence at inference, which prevents training/serving skew. A list of UDFs is not a managed, fitted pipeline, so the learned state (such as StringIndexer mappings) is not saved together. A Databricks Job orchestrates tasks and does not produce one reusable fitted model. A Delta table stores data and does not apply transformations at inference.