PyTorch Tutorial
Narges Honarvar Nazari
January 30th
Outline
• Introduction
• Training a Model in Pytorch
1. Create a Model
2. Load Data
3. Iterate Over Data and Train Model
• Test the Trained Model in PyTorch
• Transfer Learning
2
Introduction
3
What is PyTorch?
• It’s a Python-based scientific computing package
targeted at two sets of audiences 1:
• A replacement for NumPy to use the power of
GPUs
• A deep learning research platform that provides
maximum flexibility and speed
1. https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html
4
What is Tensor in PyTorch?
• A PyTorch Tensor is basically the same as a numpy
array: it does not know anything about deep
learning or computational graphs or gradients, and
is just a generic n-dimensional array to be used for
arbitrary numeric computation 1.
• The biggest difference between a numpy array and
a PyTorch Tensor is that a PyTorch Tensor can run
on either CPU or GPU. To run operations on the
GPU, just cast the Tensor to a cuda datatype 1.
1. https://pytorch.org/tutorials/beginner/examples_tensor/two_layer_net_tensor.html
5
Training a Model in PyTorch
6
Load Required Classes and
Modules
To use the Torch in Python
To create a model by layers
To set the optimization
To manipulate arrays
To Process the data
and use the existing
Models
To save the best model and get data files
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
7
Image Transformation
• We need to transform the images:
1. Change size of all images to a unanimous value.
2. Convert them to tensor. Tensor transfer the values
from scale 0-255 to 0-1.
3. Normalize the image with mean and standard
deviation for RGB values.
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 8
Image Normalization
• After converting image to tensor, every pixel value is in
range [0,1]
• Then For every pixel, we apply the following formula to
each of the channel values and range will be [-1,1]:
Pixel_Red_Value - Red_Channel_Mean Pixel_Red_Value - 0.486
Red_Channel_Standard_Deviation 0.229
Pixel_Green_Value - Green_Channel_Mean Pixel_Green_Value - 0.456
Green_Channel_Standard_Deviation 0.224
Pixel_Blue_Value - Blue_Channel_Mean Pixel_Blue_Value - 0.406
Blue_Channel_Standard_Deviation 0.225
9
Why Image Normalization?
• In general , in order to handle noise in data, data can be
transformed globally to change the scale or range of data
(normalize).1
• In Convolutional Neural Network if we don’t scale (normalize)
the values, the range of different features (e.g. image
channels) will be different.2
• Since the values are multiplied by learning rate, the features
that have larger scale might be over- compensated and
features with smaller scale might be under-compensated.2
1. https://www.coursera.org/lecture/data-genes-medicine/data-normalization-jGN7k
2. https://stats.stackexchange.com/questions/185853/why-do-we-need-to-normalize-the-
10
images-before-we-put-them-into-cnn
More Data Preprocessing
• In addition to the mentioned data
preprocessing, there are some transformation
that are used mainly for data augmentation:
o transforms.RandomHorizontalFlip()
o transforms.RandomResizedCrop(224)
• Data augmentation is a strategy that enables
practitioners to significantly increase the
diversity of data available for training models,
without actually collecting new data.1
1. https://bair.berkeley.edu/blog/2019/06/07/data_aug/
11
Mini Batch and Epoch
• Batch: Number of images which is propagated to a model
iteration.
• Epoch: An epoch refers to one cycle through the full training
dataset.1
• Example:
Number of Images = 1024
Batch Size = 4
Number of Iterations in Every Epoch: 256
1. https://deepai.org/machine-learning-glossary-and-terms/epoch
12
Load Data and Set Device
Dataset Directory
Load Data
Get number of images
and name of classes
Set the Device to GPU or CPU
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 13
Sample Network
• Here is an example of a PyTorch model
Define the layers of model (1)
Forward function is called during
forward pass (2)
Code Reference:
https://github.com/pytorch/tutorials/blob/master/beginner_source/blitz/neural_networks
14
Visualization of Sample Network
Layers which have been
declared in model initialization
(1)
conv1
pool
conv2
fc1
fc2
fc3
15
Input Image
conv1
pool
ReLu
conv2
Forward Pass of model (2) pool
ReLu
Apply View
fc1
fc2
fc3
Classification Result 16
Before Start Training
• For starting the training process we need to
1. Initialize an instance from the model which we have
already defined
2. Specify the criterion (loss) for evaluation of model
3. Specify the setting of optimizer
4. Specify the way learning rate changes during
training
1
2
3
4
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 17
Save the Best Model Parameter
• We need to train the network for the specified number
of epochs.
• Before training process, we save the initial weight as the
best model weight and set the best accuracy as zero.
• In every epoch and after finishing the training process,
we use the trained model to select the model which has
best performance on the validation set.
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
18
Iterate Over Train and Validation
Sets in every Epoch
• In every epoch we either train the model or just use
it for evaluation.
• For training, we need to set the model to train mode
and for test we need to set to eval mode.
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 19
Iterate Over every Minibatch
• We use the data loader which we have created in
previous slides to go thorough the data.
• What we get from data loader are tensors for images
(inputs) and labels and we need to transfer them to
the device which we have created before.
• Note: Phase here is ‘train’ and ‘test’
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 20
Prediction and Back Propagation
Zero the gradient before start of a new mini batch
Apply Forward Function and get logit
Get the highest logic as prediction
Compute the loss based on predicted value
Back propagate if we are in train phase
Sum the loss of batch with
all loss values
Sum correctly predicted values
in batch with all loss values
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 21
Finish Iterating over Data in One Epoch
• When iteration over all data finished then we need to
compute the loss and save the best model.
Scheduler setting (e.g. learning rate) needs to be updates
Loss and accuracy needs to be computed at the
end of epoch
Save the best model
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 22
Test on the Best Model Weight
23
Load Data for Test
Transform the test images
Load the data and get the dataset size
24
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
Test the Loaded Data
Set the model in evaluation mode
Iterate over test
data and compute
loss and correctly
predicted values
Compute the loss and
Accuracy over all data
25
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
Transfer Learning
26
Transfer Learning
• Most of the time instead of creating and
training a model from scratch, we can use the
state of art models which have been trained
on large dataset like ImageNet.
• This process is called transfer learning.
27
Transfer Learning
Figure Reference: https://indico.io/blog/exploring-computer-vision-transfer-learning/
28
Transfer Learning Example
• Let’s say we have a new dataset which has two
classes.
• We want to fine-tune the state of art model
residual net which has been trained on
ImageNet on our dataset.
Get the pre trained ResNet
Get the number of features in last layer
Replace the last fully connected layer
with a new layer
Code Reference: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 29
