1: Model Development

Learning Objectives

By the end of this chapter, you will be able to:

• Identify when SparkML is the correct choice versus single-node frameworks based on data volume, model class, and inference latency requirements
• Construct end-to-end SparkML Pipelines including the appropriate sequence of transformers, estimators, and MLflow logging
• Apply the correct SparkML estimator and transformer for tabular, text, and categorical workloads
• Tune SparkML models with CrossValidator, TrainValidationSplit, and Optuna SparkTrials integrated with MLflow
• Evaluate SparkML models with the appropriate evaluator and resolve point-in-time correctness pitfalls
• Score SparkML models for both batch and Structured Streaming inference paths
• Decide between SparkML and single-node models when targeting batch, streaming, or real-time inference
• Use nested MLflow runs, custom artifacts, and the PyFunc flavor for advanced experimentation
• Configure Feature Engineering in Unity Catalog with offline tables, online tables, on-demand features, and the FeatureEngineering Client

Executive Summary

• SparkML is the correct tool when training data does not fit a single executor or when the eventual inference surface is itself a Spark job. For single-node estimators (sklearn, XGBoost, LightGBM), Pandas Function APIs and Pandas UDFs distribute training and inference across the cluster without porting code to Spark.
• The MLflow Tracking + Unity Catalog model registry stack is the lifecycle backbone. Nested runs, custom artifacts, and the PyFunc flavor allow advanced workflows to fit inside the same registry surface that production deployments read from via aliases such as @champion and @challenger.
• Feature Engineering in Unity Catalog separates offline Delta tables (training, batch inference) from online tables (real-time serving). Point-in-time joins via timestamp_lookup_key prevent label leakage during training.
• Optuna with SparkTrials is the preferred distributed tuning approach for single-node estimators; Ray on Databricks is the preferred approach when the search itself needs distributed state, richer schedulers, or model-parallel training.

Assumptions

• Workspaces are Unity Catalog enabled, and the operating context is a UC catalog and schema (fictional catalog acme_ml, schema model_dev throughout)
• The cluster runs a Databricks Runtime ML release that supports the MLflow client with Unity Catalog as the default registry
• All code samples are Python; SQL appears only where it is the idiomatic surface for data manipulation
• Legacy assets (Workspace Model Registry, MLflow stages, dbx, MLflow Recipes) are mentioned only where the exam contrasts them with the supported path