Kapil Kumar Nagwanshi

PhD (CSE), Sr. Member IEEE, LMCSI, MIAENG

Associate Professor (CSE), ASET

Amity University Rajasthan, Jaipur
▪ Machine learning models are basically mathematical functions that represent the relationship between different
aspects of data. For instance, a linear regression model uses a line to represent the relationship between “features”
and “target.” The formula looks like this:
𝑦 = 𝑤 𝑇𝑥
▪ where x is a vector that represents features of the data and y is a scalar variable that represents the target (some
numeric quantity that we wish to learn to predict).
▪ This model assumes that the relationship between x and y is linear.
▪ The variable w is a weight vector that represents the normal vector for the line; it specifies the slope of the line.
▪ This is what’s known as a model parameter, which is learned during the training phase.

▪ “Training a model” involves using an optimization procedure to determine the best model parameter that “fits” the
data. 𝑛

𝑦 = ෍ 𝑤𝑖 𝑥𝑖
𝑖=0
▪ So a model parameter is a configuration variable that is internal to the model and whose value can be estimated
from the given data.
▪ They are required by the model when making predictions.
▪ Their values define the skill of the model on your problem.
▪ They are estimated or learned from data.
▪ They are often not set manually by the practitioner.
▪ They are often saved as part of the learned model.
▪ In statistics, hyperparameter is a parameter from a prior distribution; it captures the prior belief before
data is observed.
▪ In any machine learning algorithm, these parameters need to be initialized before training a model.
▪ These are values that must be specified outside of the training procedure.
▪ Vanilla linear regression doesn’t have any hyperparameters.
▪ But variants of linear regression do.
▪ Ridge regression and lasso both add a regularization term to linear regression; the weight for the
regularization term is called the regularization parameter.
▪ Decision trees have hyperparameters such as the desired depth and number of leaves in the tree.
Support vector machines (SVMs) require setting a misclassification penalty term.
▪ Kernelized SVMs require setting kernel parameters like the width for radial basis function (RBF)
kernels.
▪ Model Hyperparameters are the properties that govern the entire training process. The below are the
variables usually configure before training a model.
• Learning Rate
• Number of Epochs
• Hidden Layers
• Hidden Units
• Activations Functions
▪ Hyperparameters are important because they directly control the behavior of the training
algorithm and have a significant impact on the performance of the model is being trained.
▪ “A good choice of hyperparameters can really make an algorithm shine”.
▪ Choosing appropriate hyperparameters plays a crucial role in the success of our neural network
architecture. Since it makes a huge impact on the learned model.
▪ For example,
▪ if the learning rate is too low, the model will miss the important patterns in the data.
▪ If it is high, it may have collisions.

▪ Choosing good hyperparameters gives two benefits:

▪ Efficiently search the space of possible hyperparameters
▪ Easy to manage a large set of experiments for hyperparameter tuning.
The process of finding most
optimal hyperparameters in
machine learning is called Common algorithms include:
hyperparameter
optimisation.
• Grid Search
• Random Search
• Bayesian Optimisation
 ▪ Grid search is a very traditional technique for implementing
hyperparameters. It brute force all combinations. Grid search
requires to create two set of hyperparameters.
▪ Learning Rate
▪ Number of Layers

▪ Grid search trains the algorithm for all combinations by using the two
set of hyperparameters (learning rate and number of layers) and
measures the performance using “Cross Validation” technique.
▪ This validation technique gives assurance that our trained model got
most of the patterns from the dataset.
▪ One of the best methods to do validation by using “K-Fold Cross
Validation” which helps to provide ample data for training the model
and ample data for validations.
▪ The Grid search method is a simpler algorithm to use but it suffers if
data have high dimensional space called the curse of dimensionality.

https://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf
▪ It is difficult to manually change the hyperparameters and fit them on my training data every
time. Here’s why:
• it is time-consuming
• it is hard to keep track of hyperparameters we tried and we still have to try

▪ So, I quickly asked Google if there was any solution to my problem and Google showed me
something called GridSearchCV from Sklearn. Let me share how I took advantage of this
GridSearchCV to solve my problem with a simple example.
▪ GridSearchCV is a library function that is a member of sklearn’s model_selection package. It
helps to loop through predefined hyperparameters and fit your estimator (model) on your
training set. So, in the end, you can select the best parameters from the listed
hyperparameters.
▪ In addition to that, you can specify the number of times for the cross-validation for each set
of hyperparameters.
 1. estimator: estimator object you created
2. params_grid: the dictionary object that holds the hyperparameters you want
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier to try
kn = KNeighborsClassifier() 3. scoring: evaluation metric that you want to use, you can simply pass a valid
string/ object of evaluation metric
params = { 4. cv: number of cross-validation you have to try for each selected set of
'n_neighbors' : [5, 25], hyperparameters
'weights': ['uniform', 'distance’],
5. verbose: you can set it to 1 to get the detailed print out while you fit the
'algorithm': ['auto', 'ball_tree', 'kd_tree’,
'brute’] data to GridSearchCV
} 6. n_jobs: number of processes you wish to run in parallel for this task if it -1 it
will use all available processors.
grid_kn = GridSearchCV(estimator = kn,
param_grid = params,
coring = 'accuracy’, That is all pretty much you need to define. Then you have to fit your training
cv = 5, data as you do normally. You will get the first line printed like this:
verbose = 1, Fitting 5 folds for each of 16 candidates, totalling 80 fits
n_jobs = -1) ...
... • Are you confused what it means?
grid_kn.fit(X_train, y_train) ... • Simple! Since we have to try two options for n_neighbors, two
for weights and four for algorithms, altogether there are 16
different combinations we should try out.
• And for each combination, we have 5 CV fits, so 80 different fits
will be tested by our GridSearcCV object.

The time for this fit depends on the number of hyperparameters you are trying out. Once everything is finished, you will get
an output like this: [Parallel(n_jobs=1)]: Done 80 out of 80 | elapsed: 74.1min finished]
Then to know what are the best parameters you can simply print it with

# extract best estimator

print(grid_kn.best_estimator_)

Output:
KNeighborsClassifier(algorithm='auto',
leaf_size=30, metric='minkowski',metric_params=None, n_jobs=-1,
n_neighbors=25, p=2, weights='distance’)

# to test the bestfit

print(grid_kn.score(X_test, y_test))

Output:
0.9524753
▪ Randomly samples the search space and evaluates sets from a
specified probability distribution.
▪ For example, Instead of trying to check all 100,000 samples, we can
check 1000 random parameters.
▪ Drawback
▪ However, it doesn’t use information from prior experiments to select the next set
and also it is very difficult to predict the next of experiments.
▪ Hyperparameter setting maximizes the performance of the model on a validation set.

▪ Machine learning algorithms frequently require to fine-tuning of model hyperparameters.

▪ Unfortunately, that tuning is often called as ‘black function’ because it cannot be written into a
formula since the derivates of the function are unknown.
▪ Much more appealing way to optimize and fine-tune hyperparameters are enabling
automated model tuning approach by using Bayesian optimization algorithm.
▪ The model used for approximating the objective function is called surrogate model.
▪ A popular surrogate model for Bayesian optimization is Gaussian process (GP).

▪ Bayesian optimization typically works by assuming the unknown function was sampled from a
Gaussian Process (GP) and maintains a posterior distribution for this function as observations
are made.
▪ There are two major choices must be made when performing Bayesian
optimization.
▪ Select prior over functions that will express assumptions about the function being
optimized. For this, we choose Gaussian Process prior
▪ Next, we must choose an acquisition function which is used to construct a utility function
from the model posterior, allowing us to determine the next point to evaluate.
▪ A Gaussian process defines the prior distribution over functions which can be
converted into a posterior over functions once we have seen some data.
▪ The Gaussian process uses Covariance matrix to ensure that values that are close
together.
▪ The covariance matrix along with a mean µ function to output the expected value
ƒ(x) defines a Gaussian process.
▪ 1. Gaussian process will be used as a prior for Bayesian inference
▪ 2. To computing the posterior is that it can be used to make predictions for unseen test cases.
▪ Acquisition Function
▪ Introducing sampling data into the search space is done by acquisition functions. It helps to
maximize the acquisition function to determine the next sampling point. Popular acquisition
functions are
• Maximum Probability of Improvement (MPI)
• Expected Improvement (EI)
• Upper Confidence Bound (UCB)
▪ The Expected Improvement (EI) function seems to be a popular one. It is defined as
▪ EI(x)=𝔼[max{0,ƒ(x)−ƒ(x̂ )}]

▪ where ƒ(x̂ ) is the current optimal set of hyperparameters. Maximising the hyperparameters will
improve upon ƒ.
1. EI is high when the posterior expected value of the loss µ(x) is higher than the current best value ƒ(x̂)
2. EI is high when the uncertainty σ(x)σ(x) around the point xx is high.
• “Random Search for Hyper-Parameter Optimization.” James Bergstra and Yoshua Bengio. Journal of
Machine Learning Research, 2012.
• “Algorithms for Hyper-Parameter Optimization.” James Bergstra, Rémi Bardenet, Yoshua Bengio,
and Balázs Kégl.” Neural Information Processing Systems, 2011. See also a SciPy 2013 talk by the
authors.
• “Practical Bayesian Optimization of Machine Learning Algorithms.” Jasper Snoek, Hugo Larochelle,
and Ryan P. Adams. Neural Information Processing Systems, 2012.
• “Sequential Model-Based Optimization for General Algorithm Configuration.” Frank Hutter, Holger
H. Hoos, and Kevin Leyton-Brown. Learning and Intelligent Optimization, 2011.
• “Lazy Paired Hyper-Parameter Tuning.” Alice Zheng and Mikhail Bilenko. International Joint
Conference on Artificial Intelligence, 2013.
• Introduction to Derivative-Free Optimization (MPS-SIAM Series on Optimization). Andrew R. Conn,
Katya Scheinberg, and Luis N. Vincente, 2009.
• Gradient-Based Hyperparameter Optimization Through Reversible Learning. Dougal Maclaurin,
David Duvenaud, and Ryan P. Adams. ArXiv, 2015.