Spark ML
Table of Content
● Intro to Spark MLlib
● Spark ML Pipeline
● Spark ML Component Flow
● Spark ML Data Types
● Spark ML Transformers
● Understanding Outputs
● Spark ML Algorithms
● Building Pipelines
● Model Persistence
Intro to Spark ML
Spark MLlib
● Spark MLlib is Apache Spark’s Machine Learning library
● It consists of algorithms like:
○ Classification
○ Regression
○ Clustering
○ Dimensionality Reduction
○ Collaborative Filtering
Spark MLlib
Algorithms Featurization
● Regression ● Feature Extraction
● Classification ● Feature Selection
● Clustering ● Transformation
● Collaborative Filtering ● Dimensionality Reduction
Utilities Pipeline
● Linear Algebra ● Pipeline Construction
● Statistics ● Model Tuning
● Data Handling ● Model Persistence
Spark MLlib and ML
• There are two machine learning implementations in Spark (ML and MLlib):
○ spark.mllib :- ML package built on top of the RDD API
○ spark.ml :- ML package built on top of higher-level DataFrame API
● Using spark.ml is recommended because the DataFrame API is more versatile and
flexible
“Spark ML” is not an official name but used to refer to the MLlib
DataFrame-based API (spark.ml)
Spark ML Pipeline
Spark ML Pipeline
• In machine learning, there are a lot of transformation steps that are performed to pre-process
the data
• We repeat the same steps while making prediction
• You may often get confused about this transformations while working on huge projects
• To avoid this, pipelines were introduced that hold every step that is performed to fit the data
on a model
Spark ML Pipeline
• The Pipeline API in Spark chains multiple Transformers and Estimator specifying a ML workflow
• It is a high-level API for MLlib that lives under the spark.ml package
Pipeline
Transformer x Transformer y … Transformer z Estimator
Dataframe Dataframe
Transformers
• A Transformer takes a dataset as input and produces an augmented dataset as output
• It basically transforms one DataFrame into another DataFrame
Transformer
Estimators
• An Estimator fit on the input data that produces a model
• For eg., logistic regression is an Estimator that trains on a dataset with labels and
features and produces a logistic regression model
Estimator Transformer
• The model acts as a Transformer that transforms the input dataset
For eg., logistic regression model can later be used to make predictions which
technically adds prediction columns (Transformation) in the dataset
Spark ML Component Flow
Spark ML Component Flow
• Pipeline API chains Transformers and Estimator each as a stage to specifying ML
workflow
• These stages are run in order
• The input DataFrame is transformed as it passes through each stage
• Evaluator then evaluates the model performance
Spark ML Component Flow
Load Data
Pipeline
Min-Max Scalar
One Hot Encoder Transformers
Vector Assembler
Logistic Regression Estimator
Evaluate
Spark ML Component Flow
Load Data Load Data
Min-Max Scalar Min-Max Scalar
One Hot Encoder One Hot Encoder
Vector Assembler Vector Assembler
Logistic Regression Logistic Regression
Evaluate Evaluate
Pipeline of machine learning components Reusing the pipeline on new data
Spark ML Component Flow
• Spark ML algorithms (estimators) expects all features to be contained within a
single column in the form of Vector
• It is one of the important spark ml data types that you need to understand before we
take a look at different feature transformers
Spark ML Data Types
Spark ML Data Types
• Spark ML uses the following data types internally for machine learning algorithms
○ Vectors
○ Matrix
Spark ML Data Types
● These data types help you for a process called featurization
● Conversion of numerical value, string value, character value, categorical value into
numerical feature is called featurization
● The data once converted to these data types can be further passed to the ML
algorithm in Spark
Spark ML Data Types
● For eg.: Consider the following sentences
○ I love programming
○ Python is a programming language
○ Python is my favourite programming language
○ Data science using Python
Spark ML Data Types
● Make list of the word such that one word should be occurring only once, then the list
looks like as follow:
[“I”, “love”, “programming”, “Python”, “is”, “a”, “language”, “my”, “favourite”,
“Data”, “Science”, “using”]
● Now count occurrence of word in a sentence with respect to this list
Spark ML Data Types
● For example- vector conversion of sentence “Data science using Python” can be
represented as :
“I” - 0
“love”- 0
“programming” - 0
“Python” - 1
“is” - 0
“a” - 0
“language” - 0
“my” - 0
“favourite” - 0
“Data” - 1
“Science” - 1
“using” - 1
Spark ML Data Types
● By following same approach other vector value are as follow:
I love programming = [1 1 1 0 0 0 0 0 0 0 0 0]
Python is a programming language = [0 0 1 1 1 1 1 0 0 0 0 0]
Python is my favourite programming language = [0 0 1 1 1 0 1 1 1 0 0 0]
Data science using Python = [0 0 0 1 0 0 0 0 0 1 1 1]
Spark ML Data Types
● And the sentences can also be converted into 4*12 matrix
array([[1 1 1 0 0 0 0 0 0 0 0 0],
[0 0 1 1 1 1 1 0 0 0 0 0],
[0 0 1 1 1 0 1 1 1 0 0 0],
[0 0 0 1 0 0 0 0 0 1 1 1]])
● Now that you have understood vectors and matrix, let us now focus on the different
types of vectors
Please Note
The elements of vectors and matrix are NOT always 0s and 1s.
Spark ML Data Types - Local Vector
● A local vector has integer-typed and 0-based indices and double-typed values
● They are stored on local machine
● A local vector can be represented as:
○ Dense Vector
○ Sparse Vector
Please Note
Dense and Sparse Vectors are vector representation of data
Spark ML Data Types - Local Vector
● Dense Vectors:
○ By definition, dense means closely compacted
○ Dense Vector is a vector representation that contains many values or values that
are not zeros (very few zero values)
○ It is basically an array of values
○ For eg. A vector (3.0, 5.0, 8.0, 0.0) can be represented in dense format as [3.0,
5.0, 8.0, 0.0]
Spark ML Data Types - Local Vector
● Dense Vectors PySpark
Spark ML Data Types - Local Vector
● Sparse Vectors:
○ By definition, sparse means thinly dispersed or scattered
○ If a vector has a majority of its elements as zero, it can be represented as sparse
vector
○ It stores the size of the vector, an array of indices, and an array of values
corresponding to those indices
○ For ex. A vector (0.0, 3.0, 0.0, 8.0, 0.0) can be represented in sparse format as
[5, [1,3], [3.0, 8.0]]
Spark ML Data Types - Local Vector
[5, [1,3], [3.0, 8.0]]
Total no. of elements Array of indices where non-zero Array of values corresponding to
(integer typed) elements are present (integer each index (double-typed)
typed)
Parallel Arrays
● A sparse vector is used for storing non-zero entries for saving space
Spark ML Data Types - Local Vector
● Sparse Vector PySpark
In computer science, Parallel Array is an implicit data structure that contains multiple
arrays
Each of these arrays are of the same size and the array elements are related to each other
i-th element of each array is closely related and all i-th elements together represent an
object or entity
Spark ML Data Types - Labeled Point
● Labeled Point is a type of local vector
● It can either be dense or sparse
● It is associated with label/response variable
● Used in supervised learning algorithms
● A label should either be 0 (-ve) or 1 (+ve) for binary classification
● A label should be class indices starting from zero: 0, 1, 2,…
Spark ML Data Types - Labeled Point
Type Label Values
Regression Decimal Values
Binary Classification 0 or 1
Multi-class Classification 0,1,2,3, ….
Spark ML Transformers
ML Feature Transformers
● Feature building is a super important step for model building
● Some of the common feature transformer that we use for model building are:
○ Binarizer ○ VectorAssembler
○ Bucketizer ○ VectorIndexer
○ StringIndexer ○ StandardScaler
○ IndexToString ○ MinMaxScaler
○ OneHotEncoder
● Most Transforms are under org.apache.spark.ml.feature package
ML Feature Transformers
● Binarizer
○ Binarization is used for thresholding numerical feature to binary feature (0 or 1)
○ Binarizer takes inputCol, outputCol and threshold for binarization as parameter
○ Values greater than the threshold value are binarized to 1.0
○ Values less than the threshold value are binarized to 0.0
ML Feature Transformers
● Consider the following dataframe:
We can create a new variable “BodyType” by binarizing the 'BMI' variable (1- obese and 0- healthy)
If your BMI is 30.0 or higher, the BodyType falls in the obese range)
ML Feature Transformers
● Code:
Value equal to or above the threshold value 30 is set to 1 (denoting
obesity) in the new column ‘BodyType’
ML Feature Transformers
● Bucketizer
○ Bucketization is used for creating group of values of a continuous feature
○ Bucketizer takes inputCol, outputCol and splits for mapping continuous
features into buckets as parameter
○ There are n buckets for n+1 splits
○ The splits that you provided have to be in strictly increasing order, i.e. s0 < s1
< s2 < ... < sn
ML Feature Transformers
● Code:
You can check there are 4 (0,1,2,3) buckets for 5 splits
ML Feature Transformers
● StringIndexer
○ StringIndexer converts a string column to an index column
○ The most frequent label gets index 0
○ Labels are basically ordered by their frequencies
Please Note
There can be a situation when the StringIndexer may encounter a new label
This usually happens when you fit StringIndexer on one dataset and then use it to transform
incoming data that may have a new label
You can use any of the following three strategies to handle the situation by setting the parameter
setHandleInvalid to:
○ ‘error’: throw an exception (which is the default)
○ ‘skip’: skip the row containing the unseen label entirely
○ ‘keep’: put unseen labels in a special additional bucket, at index numLabels
ML Feature Transformers
● Code:
ML Feature Transformers
● Output:
ML Feature Transformers
● IndexToString
○ IndexToString converts a column of label indices back to a column containing
the original labels as strings
○ It is like the inverse of StringIndexer: You can retrieve the labels that were
transformed by StringIndexer
○ This transformer is mostly used after training a model where you can retrieve
the original labels from the prediction column
Class Exercise
Use IndexToString to convert index column into it respective string value
ML Feature Transformers
● OneHotEncoderEstimator
○ OneHotEncoderEstimator converts the label indices to binary vector
representation with at most a single one-value
○ It represents the presence of a specific feature value from among the set of all
feature values
○ It encodes the features into a sparse vector
ML Feature Transformers
● Code:
ML Feature Transformers
● Output:
Please Note
One hot encoder in spark work very differently than the way it works in sklearn (like
dummy column creation style)
Only one feature column is created representing categorical indices in the form of sparse
vector in each row
You may want to convert this sparse vector to dense vector later for scaling, if required
Please Note
It is primarily used for linear model (ex. Logistic Regression) to encode categorical features
since these algorithms expect continuous features
Such representations proves to be inefficient to be used with algorithms which handle
categorical features intrinsically
ML Feature Transformers
● VectorAssembler
○ MLlib expects all features to be contained within a single column
○ VectorAssembler combines multiple columns and gives single column as
output
○ The output column represents the values for all of the input columns in the
form of vector (DenseVector or SparseVector depending on which use the least
memory)
ML Feature Transformers
● Code:
ML Feature Transformers
● Output:
If you notice, the feature column contains sparse vector
Please Note
VectorAssembler chooses dense vs sparse output format based on whichever one
uses less memory
It does not convert the vector into a dense vector during the merging process
You may want to convert this feature vector, if sparse, into a dense vector to perform
scaling
ML Feature Transformers
● VectorIndexer
○ VectorIndexer automatically identifies the categorical features from the feature
vector (output from VectorAssembler)
○ It then indexes categorical features inside of a Vector
○ It is the vectorized version of StringIndexer
○ This step is mostly used after the VectorAssembler stage
ML Feature Transformers
● Code:
ML Feature Transformers
● Output:
ML Feature Transformers
Using the StringIndexer output directly as a feature will not make sense because it converts
the categorical variable into nominal variable (do not have any order). Hence we one hot
encode them
The VectorIndexer does the same but in the backend
Please Note
VectorIndexer let us skip the one hot encoding stage for encoding the categorical features
As discussed earlier, we should not use one hot encoding on categorical variables for
algorithms like decision tree and tree ensembles
VectorIndexer are chosen over OneHotEncoderEstimator in such scenario which allows
these algorithms to treat categorical features appropriately
ML Feature Transformers
● StandardScaler
○ StandardScaler scales each value in the feature vector such that the mean is 0
and the standard deviation is 1
○ It takes parameters:
■ withStd: True by default. Scales the data to unit standard deviation
■ withMean: False by default. Centers the data with mean before scaling
Please Note
To use scaling transformers, we need to assemble the features into a feature vector first
(using VectorAssembler)
They do not convert sparse vector to dense vector internally. Therefore, it is very important
to convert the sparse vector to a dense vector before running this step to avoid incorrect
results as it does not throw error for the input sparse vector
ML Feature Transformers
● Code: We first convert sparse vector into dense vector
ML Feature Transformers
● Output:
ML Feature Transformers
● Code: We then apply StandardScaler on the dense vector
ML Feature Transformers
● Output:
ML Feature Transformers
● MinMaxScaler
○ MinMaxScaler scales each value in the feature vector between 0 and 1
○ Though (0, 1) is the default range, we can define our range of max and min
values as well
○ It takes parameters:
■ min: 0.0 by default. Lower bound value
■ max: 1.0 by default. Upper bound value
Class Exercise
Use MinMaxScaler to scale the dense features
ML Feature Transformers
● Normalizer
○ Normalizer normalize each value in the feature vector to have unit norm
○ It takes parameter p which specifies p-norm used for normalization. By
default, the value of p is 2
Class Exercise
Use Normalizer to scale the dense features
Understanding Outputs
Understanding Output of a Model
• After you transform the dataframe with the model that you built, it may add
additional columns as predictions depending upon the algorithm:
○ rawPrediction
○ probability
○ prediction
Understanding Output of a Model
• rawPrediction
- It stores the raw output of a classifier for each possible target variable label
- The meaning of a “raw” prediction may vary between algorithms
- It gives a measure of confidence in each possible label (where larger = more
confident)
- For eg., for logistic regression the rawPrediction is calculated with the help of logit
Understanding Output of a Model
• probability
- It stores the probability of a classifier for each possible target variable label given
the raw prediction
- For eg., In logistic regression, probability is the result of applying the logistic
function ( exp(x)/(1+exp(x)) ) to rawPrediction
Understanding Output of a Model
• prediction
- It is the corresponding class that the model has predicted for given probability array
- It takes the maximum value out of the probability array, and it gives the most
probable label (single number)
Interpretation
Probability for class 0 is greater
rawPrediction rawPrediction probability for probability for
for class 0 for class 1 class 0 class 1
Probability for class 1 is greater
Spark ML Algorithms
Spark ML Algorithms
● As discussed earlier, all spark ml model trains off only one column of data
● You should extract values from each row and pack them into a vector in a single
column named features (name not compulsory)
● Therefore, every spark ml model has ‘featureCol’ as a parameter
● Only supervised learning models will have ‘labelCol’ along with ‘featureCol’ as a
parameter
Spark ML Algorithms
Common Spark ML Parameters:
Parameter Input
Description Note
Name Type
labelCol Double Target Column Only for supervised learning
algorithms
featuresCol Vector Features Vector For all algorithms
Spark ML Algorithms Example: Logistic Regression
● We use LogisticRegression from pyspark.ml package to train (fit) Logistic Regression
with the features
● LogisticRegression.fit returns LogisticRegressionModel object
● This object acts as a transformer that add the prediction columns to the dataframe
● This is applicable to all the spark ml algorithms
Spark ML Algorithms Example: Logistic Regression
● Code: Logistic Regression pyspark ml implementation
Spark ML Algorithms Example: Logistic Regression
● Output
Interpretation
• rawPrediction: it is the raw output of the logistic regression classifier (array with
length equal to the number of classes)
• probability: it is the result of applying the logistic function to rawPrediction (array
of length equal to that of rawPrediction)
• prediction: it is the argument where the array probability takes its maximum value,
and it gives the most probable label (single number)
Logistic Regression Model Evaluation
• Spark ML provides a suite of metrics for the purpose of evaluating the performance
of machine learning models
• Let us evaluate the logistic regression model that we built using
BinaryClassificationEvaluator
Logistic Regression Model Evaluation
● Code: Evaluating model performance using BinaryClassificationEvaluator
Model Evaluation
● You can also use model.summary for logistic regression to get the performance metrics
Model Evaluation
● Output
Spark ML Algorithms
Algorithm Spark ML Package Spark ML Sklearn Equivalent Output
Parameter(s)
Linear
pyspark.ml.regression LinearRegression LinearRegression predictionCol
Regression
rawPredictionCol
Logistic
pyspark.ml.classification LogisticRegression LogisticRegression probabilityCol
Regression
predictionCol
rawPredictionCol
Decision Tree
pyspark.ml.classification DecisionTreeClassifier DecisionTreeClassifier probabilityCol
Classification
predictionCol
Decision Tree
pyspark.ml.regression DecisionTreeRegressor DecisionTreeRegressor predictionCol
Regression
Spark ML Algorithms
Algorithm Spark ML Package Spark ML Sklearn Equivalent Output
Parameter(s)
rawPredictionCol
Random Forest
pyspark.ml.classification RandomForestClassifier RandomForestClassifier probabilityCol
Classification
predictionCol
Random Forest
pyspark.ml.regression RandomForestRegressor RandomForestRegressor predictionCol
Regression
Gradient rawPredictionCol
Boosted Trees pyspark.ml.classification GBTClassifier GradientBoostingClassifier probabilityCol
Classification predictionCol
Gradient
Boosted Trees pyspark.ml.regression GBTRegressor GradientBoostingRegressor predictionCol
Regression
Spark ML Algorithms
Algorithm Spark ML Package Spark ML Sklearn Equivalent Output
Parameter(s)
Support Vector rawPredictionCol
pyspark.ml.classification LinearSVC
Machines LinearSVC probabilityCol
(SVM) predictionCol
rawPredictionCol
Naive Bayes pyspark.ml.classification NaiveBayes GaussianNB probabilityCol
predictionCol
K-means pyspark.ml.clustering KMeans GradientBoostingClassifier predictionCol
Model Evaluation
● Following evaluators are available in pyspark.ml.evaluation package
Evaluator Metric Available
BinaryClassificationEvaluator areaUnderROC
areaUnderPR
MulticlassClassificationEvaluator
f1, accuracy, weightedPrecision,
weightedRecall, weightedTruePositiveRate,
weightedFalsePositiveRate, weightedFMeasure,
truePositiveRateByLabel, falsePositiveRateByLabel,
precisionByLabel,
recallByLabel, fMeasureByLabel, logLoss, hammingLoss
Model Evaluation
Evaluator Metric Available
RegressionEvaluator rmse, mse, r2, mae, var
MultilabelClassificationEvaluator
subsetAccuracy, accuracy, hammingLoss,
precision, recall, f1Measure,
precisionByLabel, recallByLabel, f1MeasureByLabel,
microPrecision, microRecall, microF1Measure
ClusteringEvaluator silhouette
Building Pipeline
Building Spark ML Pipeline
● As discussed earlier, a spark pipeline is a sequence of Transformers and an Estimator
● These stages run in order and the dataframe is transformed as it passes through each
stage
● We will now see how to build a pipeline in pyspark
Building Spark ML Pipeline
● To build a pipeline we import the Pipeline module from pyspark.ml package
● Next, we create a pipeline object by passing all transformers and an estimator as a
list of stages
● This object is later fit on the raw training set, which creates a pipeline model
● This model is later used as a transformer to be applied on testing set to make
predictions
Building Spark ML Pipeline
● Code: Building and implementing a spark ml pipeline
Building Spark ML Pipeline
● Output
Model Persistence
Model Persistence
● In real-life scenarios, you will be producing ML model and hands it over to the
development team for deploying in a production environment
● This becomes easier with model persistence
● Model persistence means saving your model to a disk for later use without the need
to retrain your model
Model Persistence
● We use model.save(‘path’) to save our model at the desired location
● It might happen that you wish to retrain your model and save it at the same the place
● In those cases, use model.write().overwrite().save(‘path’) to save your retrained
model at the same place
Model Persistence
● Code - Saving the model
Model Persistence
● You can then load the model and perform predictions
● Use PipelineModel module from pyspark.ml package to load the persisted pipeline
model
● The loaded model can then be used for perform prediction on test data
Model Persistence
● Code: Loading the model
Model Persistence
● Output
Summary
● Spark MLlib is Apache Spark’s Machine Learning library
● spark.mllib package built on top of the RDD API
● spark.ml package built on top of higher-level DataFrame API
● Pipeline API chains Transformers and Estimator each as a stage to specifying ML workflow
● Spark ML library provides number of transformers to preprocess the data
Thank You
