A machine learning engineer wants to apply a registered MLflow model to a large Spark DataFrame for batch scoring, distributing the computation across the cluster rather than running inference row-by-row on the driver. Which approach correctly achieves this?
- A. Register the model as a REST endpoint and issue one HTTP request per row from the driver in a sequential loop
- B. Load the model as a Spark UDF using
mlflow.pyfunc.spark_udf()and apply it to the DataFrame withwithColumn(), so predictions run in parallel across executors✓ Correct - C. Batch scoring against a Spark DataFrame is not supported by MLflow; the DataFrame must first be exported to CSV and scored by an external service
- D. Load the model with
mlflow.pyfunc.load_model()on the driver, collect the entire Spark DataFrame to a Python list, and loop over rows calling.predict()one at a time
Explanation
mlflow.pyfunc.spark_udf() wraps a registered model as a Spark UDF, which Spark then distributes across executors, letting batch inference run in parallel at scale — the standard pattern for batch-scoring a Spark DataFrame. Option 2 collects everything to the driver and loops sequentially, discarding Spark's parallelism and risking driver memory exhaustion at scale, which is exactly the anti-pattern this question is testing against. Option 3 is false — MLflow explicitly supports Spark DataFrame batch scoring via spark_udf. Option 4 turns a bulk, parallelizable batch job into a slow sequential loop of network calls, which is a poor fit for large-scale batch inference.