Department of

Computer Science and Engineering

Course Name / Code : Deep Learning / 10212CS215

Slot : S2
Category : PROGRAM ELECTIVE
Faculty Name : R.T.THIVYA LAKSHMI.,M.TECH.,(PhD),
TTS 3821
Assistant Professor, CSE

School of Computing
Vel Tech Rangarajan Dr. Sagunthala R&D Institute of
Science and Technology
 DEEP LEARNING
• Deep learning is a subset of machine learning that is
inspired by the structure and function of the human
brain.
• It uses large neural networks to process and learn
from data, enabling machines to make accurate
predictions and decisions.
 DEEP LEARNING
• Deep learning has been used in a variety of
applications, from speech recognition and
natural language processing to image and
video recognition, and even self-driving cars.
• It has the ability to learn from vast amounts of
data, making it a powerful tool for solving
complex problems.
 Unit – I Data Processing and Deep Learning Framework

• Introduction to Data Handling

• Data Processing and Normalization using Deep Learning Libraries

• Data Augmentation Techniques

• Deep Learning Frameworks – PyTorch, TensorFlow and Keras

• Deep Learning Parameters – Epoch, Learning Rate, Batch Size

• Open Source deep learning tools

and Project
Management
(SEPM)

Department of Computer Science and Engineering

 Introduction to Data Handling
 Data handling is the process of collecting, cleaning, organising and analysing data
available in a firm or from outside sources.
 Huge amounts of data, popularly called big data, are collected and analysed by
organisations to uncover valuable insights.
 These insights help the firm understand the reason for various events in the past
and predict outcomes in the future.
Types of Data:
 Data handling methods can be performed based on the types of data. The data is
classified into two types, such as:
i. Qualitative Data – Descriptive information (Eg. Colors, Flavors, Beauty,…)
ii. Quantitative Data – Numerical information (Discrete or Continuous) (Eg.
Height, Weight, Temperature,..)
1. Data Collection

 It is the first step in data handling. Data is available from various sources, and the
analysts must collect them before they can use them.

 Before they collect the data, experts must decide the desired outcome of data
analysis.

 It will help them know the kind of information they must collect and where they will
be available.

 The next step is deciding what methods will be used to collect and process the data.

 Information may be collected using various methods like surveys, transactional

tracking, interviews, observation, forms or social media monitoring.
 Two methods of data collection are popularly used.

 The primary method is the first collection of data performed by researchers before
anyone has done any process. This method is expensive but more accurate.

 In the secondary method, experts collect data that has already undergone processing.
This method is inexpensive but may need to be more accurate. It can be said to be
second-hand information.
2. Data Cleaning:

 Data cleaning or cleansing is the job of removing corrupt, incorrect,

duplicate, wrongly formatted or incomplete data from the collected data.
 This process is very important in data handling because if the information is not
correct, the outcome of the analysis will also be wrong.
 It can have serious implications for the decisions the company management
takes.
 Data cleaning should not be mistaken for transformation because that is the task
of converting data from one format to another. That process is used for
converting the format for storing or analyzing purposes.
 There are different steps in data cleaning, and they are as follows:
i. Removing Duplicate Data

 The first step in cleansing is to remove duplicate or irrelevant observations

from the data. When collecting data from various sources, data duplication
may exist.

ii. Fix Structural Errors

 These occur as naming errors, typos or wrong capitalisations. These must be

corrected as these can cause mislabelled categories or classes.

iii. Filter Unwanted Outliers

 When observing the data, you may find details that don’t fit into the data
you are analysing. Removing such information will make your outcome
more accurate. Not all outliers are incorrect, and this makes it necessary to
make the right decision here.
iv. Taking Care Of Missing Data

 Taking care of missing data is essential because most algorithms will not
accept missing values. One way to take care of this is to remove the
observation with missing values. The other way is to input the missing value
based on the other observations. It is also possible to alter the way you use
the data.
v. Validate And Quality Control
 At this stage, it is necessary to see that you can answer some questions. You
must confirm that the data makes sense and follows the rules for its field. It
is also essential to see if the information proves or disproves your theory or
brings out any insight. You must also ensure that you can find trends in the
data.
3. Data Processing:

 Data processing refers to the collection, manipulation, and analysis of raw data
to extract meaningful information and insights.
 It involves a series of steps that transform data into a more usable and valuable
form. Data processing is a fundamental aspect of many industries, including
business, finance, healthcare, research, and more.
 Stages of Data Processing are:

a. Data Collection d. Data Transformation g. Data Visualization

b. Data Entry e. Data Storage h. Data Interpretation
c. Data Cleaning f. Data Analysis i. Data Reporting
Data Processing
Data Processing
 Data processing in Deep Learning (DL) refers to the various operations and
transformations performed on input data to make it suitable for training deep neural
networks.

 It involves tasks such as data cleaning, pre-processing, normalization,

augmentation and feature extraction.

 The goal of data processing is to prepare the data in a format that can be effectively
fed into a deep learning model for training or inference.

 Here are some key steps involved in data processing for deep learning:

1. Data Cleaning: This step involves handling missing values, outliers, and any
noise in the data. It may include techniques like imputation for missing values or
removing outliers that could negatively impact the model's performance.
2. Data Preprocessing: Preprocessing steps depend on the type of data and the specific
requirements of the deep learning model. Common pre processing tasks include data
scaling, dimensionality reduction, handling categorical variables, and converting
data into a suitable format. Examples include normalizing numerical features, one-hot
encoding categorical variables, or applying text tokenization and vectorization.

3. Data Normalization: Normalizing the data brings the input features to a similar scale,
often between 0 and 1 or with a mean of 0 and standard deviation of 1.
Normalization helps in improving convergence and stability during training and
prevents certain features from dominating others.

4. Data Augmentation: Data augmentation is a technique used to artificially increase the

diversity of the training data by applying random transformations. These
transformations can include image rotation, translation, scaling, or flipping. Data
augmentation helps improve the model's ability to generalize by exposing it to a wider
range of variations in the training data.
5. Feature Extraction: In some cases, the raw input data may not be directly usable by
the deep learning model. Feature extraction involves deriving new features or
representations from the input data that capture relevant information for the task at
hand. This could be done through techniques like convolutional neural networks
(CNNs) for image data or recurrent neural networks (RNNs) for sequential
data.

6. Data Splitting: The dataset is typically split into training, validation, and test sets.
The training set is used to train the deep learning model, the validation set is used for
hyperparameter tuning and model evaluation during training, and the test set is used
for final model evaluation to assess its performance on unseen data.

 Each step-in data processing plays a crucial role in preparing the data for deep
learning tasks. The specific data processing techniques applied depend on the nature
of the data, the problem domain, and the requirements of the deep learning model
Normalization
Normalization
 Normalization in the context of deep learning techniques refers to the process of
scaling and standardizing input data to ensure that it falls within a specific range or
distribution.

 Normalizing input data is a common preprocessing step in deep learning models, as

it helps improve the convergence and stability of the training process.

Xn = (X - Xminimum) / ( Xmaximum - Xminimum)

 Case1- If the value of X is minimum, the value of Numerator will be 0; hence

Normalization will also be 0.

 Case2- If the value of X is maximum, then the value of the numerator is equal to the
denominator; hence Normalization will be 1.
Normalization
 Case3- On the other hand, if the value of X is neither maximum nor minimum, then
values of normalization will also be between 0 and 1.

 Hence, Normalization can be defined as a scaling method where values are shifted
and rescaled to maintain their ranges between 0 and 1, or in other words; it can be
referred to as Min-Max scaling technique.

 There are different types of normalization techniques commonly used in deep

learning:
i. Min-Max normalization: Also known as feature scaling, this technique rescales the
input data to a specific range, typically between 0 and 1. It is achieved by subtracting the
minimum value of the data and dividing by the difference between the maximum and
minimum values. Min-Max normalization helps ensure that all input features have a
consistent scale.
Normalization
ii. Standardization: Standardization, also known as z-score normalization, transforms
the input data to have zero mean and unit variance. It involves subtracting the mean of
the data and dividing by the standard deviation. Standardization is useful when the
input features have different scales or follow different distributions. Hence,
𝑋−𝜇
standardization can be expressed as follows: 𝑋 ′ =
𝜎

iii. Batch normalization: Batch normalization is a technique specifically designed for deep
neural networks. It normalizes the activations of each layer by computing the mean
and standard deviation of a batch of training examples. It helps address the internal
covariate shift problem by stabilizing the distribution of the inputs to each layer,
leading to faster and more stable training.

iv. Layer normalization: Similar to batch normalization, layer normalization normalizes

the activations within a layer. However, instead of computing statistics over a batch, it
computes the mean and standard deviation across the features of a single training
example. Layer normalization is beneficial for recurrent neural networks (RNNs) and
models with variable-length inputs.
 Normalization techniques are typically applied to the input data before feeding it
into a deep learning model. This ensures that the data is in a suitable range and
distribution, which can facilitate more effective learning and convergence.

 It also helps prevent certain features from dominating the learning process due to
differences in scale or magnitude.

 It's worth noting that the choice of normalization technique depends on the
specific requirements of the dataset and the characteristics of the deep learning
model being used.

 Experimentation and empirical evaluation are often necessary to determine the

most appropriate normalization approach for a given task.
Normalization using DL Libraries
 Normalization is a crucial pre processing step in deep learning, as it helps to bring
input data into a standard range and improve model convergence and performance.
Deep learning libraries provide various tools and functions for normalization. Some
examples using popular deep learning libraries such as TensorFlow and PyTorch.
1. Normalization in TensorFlow:

 TensorFlow provides several methods for normalization, including using the

tf.keras.layers.BatchNormalization layer or applying other normalization
techniques manually. Here's an example of using BatchNormalization:

import tensorflow as tf

# Assume you have input data 'x' with shape (batch_size, height, width, channels)

normalized_x = tf.keras.layers.BatchNormalization()(x)
2. Normalization in PyTorch:

 PyTorch also offers different normalization options. One common approach is to use
the torchvision.transforms.Normalize class, which can be applied to input tensors.
Here's an example:
import torch
import torchvision.transforms as transforms
# Assume you have input data 'x' as a PyTorch tensor
mean = [0.5, 0.5, 0.5] # Mean values for each channel
std = [0.5, 0.5, 0.5] # Standard deviation for each channel
transform = transforms.Compose([
transforms.ToTensor(), # Convert to tensor
transforms.Normalize(mean=mean, std=std) # Normalize
])
normalized_x = transform(x)
Data Augmentation
 Data augmentation techniques are used to increase the diversity and size of the training
dataset by applying various transformations to the input data. The goal of data
augmentation is to enhance the model's ability to generalize and improve its
performance by exposing it to a wider range of variations and scenarios.
 Image Augmentation: For image datasets, data augmentation techniques can include
operations
1. Random Flips: Horizontally or vertically flipping the image.
2. Random Rotations: Rotating the image by a random angle.
3. Random Crop: Cropping a random portion of the image.
4. Random Zoom: Zooming in or out on the image.
5. Random Brightness/Contrast: Adjusting the brightness or contrast of the image.
6. Random Hue/Saturation: Modifying the hue or saturation of the image.
7. Random Gaussian Noise: Adding Gaussian noise to the image
 Geometric Transformations:
1. Translation: Shifting the image in the x and y directions.
2. Rotation: Rotating the image by a specific angle.
3. Scaling: Rescaling the image by a factor.
4. Shearing: Distorting the image by shearing it along the x or y-axis.

 Text Data Augmentation:

1. Synonym Replacement: Replacing words with their synonyms.
2. Random Insertion: Inserting random words into the text.
3. Random Deletion: Randomly deleting words from the text.
4. Random Swap: Swapping adjacent words in the text.
5. Character-level Augmentation: Perturbing individual characters in the text.
 Audio Data Augmentation:
1. Time Shifting: Shifting the audio in time.
2. Speed Perturbation: Increasing or decreasing the playback speed.
3. Pitch Shifting: Modifying the pitch of the audio.
4. Adding Background Noise: Mixing the audio with random background noise.
5. Time Stretching: Stretching or compressing the audio in time

 Generative adversarial networks (GANs):

GANs are a type of deep learning model that can generate new synthetic data samples
by learning from existing data. GANs can be used for data augmentation by training a
generator network to create new data samples that closely resemble the original data.
This technique is particularly useful when the data distribution is complex and difficult
to model explicitly.
Deep Learning Frameworks
 A framework refers to a software library or platform that provides a collection of
tools, functions, and abstractions to simplify the development, training, and
deployment of deep neural networks.

 There are several popular deep learning frameworks available that provide tools and
libraries to build and train deep neural networks. Some of the commonly used deep
learning frameworks are:
1. TensorFlow

 Developed by Google Brain, Tensorflow is by far, one of the most used deep
learning frameworks. Tensorflow provided a wide range of APIs(application
programming language) from preprocessing to the modelling of data.

 It is written in Python, C++ and CUDA. Runs on almost all platforms – Linux,
Windows, macOS, iOS and Android. For Android and iOS, Tensorflow provides
the TensorflowLite version.

 For deployment, Tensorflow has powerful clustering support. Tensorflow can be

used with CPUs, GPUs and TPUs as well.

 Applications – Google Teachable Machine(No code ML training platform),

RankBrain(SEO), Deep Speech(speech translation), Nsynth(for making music),
Uber, Delivery Hero, Ruangguru, Hepsiburada, 9GAG, Channel.io.
2. PyTorch

 Developed by Facebook’s AI Research Lab, PyTorch is another widely used deep

learning framework mainly for its Python interface.

 PyTorch is built on top of the Torch library. PyTorch has similarities with
Tensorflow and thus in major competition with it. PyTorch was mainly developed
for research and production deployment purpose.

 Using TorchServe, PyTorch faster speed and ease of access to deploy models.

 TorchScript provides flexibility in the transition of graph mode functionalities.

Distributed training allows PyTorch to provide optimisation in models.

 Applications – NVIDIA, Apple, Robin Hood, Lyft, Ford Motor Company

3. Keras

 Cross-platform neural network library is written in Python, developed by Francis

Chollet. Keras is a high-level API built on top of Tensorflow. Keras is the most
used deep learning framework in Kaggle.

 Keras best runs on GPUs and TPUs. Keras models can easily be deployed to the
web, iOS and Android. Keras is known for its fast computation, user-friendliness
and ease of access. Keras has an active community and thus under constant
development.

 Applications – Used by scientific organisations such as NASA, CERN, NIH and

LHC, Lockheed Martin, Ford Motor Company
Deep Learning Parameters
 In deep learning, several hyperparameters need to be set before training a neural
network. Three critical hyperparameters are the number of epochs, batch size, and
learning rate.

1. Epoch:
 An epoch represents a complete pass through the entire training dataset during the
training process.

 In other words, it indicates the number of times the model sees and learns from the entire
training data. Increasing the number of epochs allows the model to potentially converge
to a better solution.

 However, setting too many epochs may result in overfitting, where the model memorizes
the training data instead of learning generalizable patterns.

 The optimal number of epochs depends on the complexity of the problem and the size of
the dataset and is typically determined through experimentation.
Deep Learning Parameters
2. Batch Size:
 During training, the dataset is divided into smaller subsets called batches. The batch size
refers to the number of samples in each batch. The choice of batch size affects both
computational efficiency and the quality of the model's parameter updates.
 Larger batch sizes tend to provide more stable gradient estimates, leading to smoother
convergence. They also offer better utilization of hardware resources, especially when
training on GPUs. However, larger batch sizes require more memory and may lead to
slower convergence or difficulties in reaching sharp minima.
 Smaller batch sizes introduce more noise into the parameter updates, but they can help the
model generalize better as they provide more diverse examples in each update. Smaller
batches also consume less memory but may result in longer training times due to
increased computational overhead.
 Selecting an appropriate batch size often involves a trade-off between convergence
speed, memory usage, and generalization performance. Common choices range from
small values (e.g., 32 or 64) to larger values (e.g., 256 or 512).
Deep Learning Parameters
3. Learning Rate:
 The learning rate determines the step size at which the model's parameters are updated
during the optimization process. It is a critical hyperparameter as it affects the
convergence speed and the quality of the learned solution.

 A high learning rate may cause the optimization algorithm to overshoot the optimal
solution, leading to unstable or divergent behavior.

 A low learning rate may result in slow convergence or getting stuck in suboptimal
solutions.

 The choice of an appropriate learning rate depends on the problem and the model
architecture. It is often set based on a combination of intuition and empirical
experimentation.

 Hyperparameter tuning, including finding the optimal number of epochs, batch size, and
learning rate, is often an iterative process. It requires experimentation, monitoring the
model's performance on validation data, and making adjustments based on observed
results.
Open Source Deep Learning Tools

Deep Learning Tools Developed as a part of

TensorFlow Google Research
Open-ended Neuro-Electronic Intelligent
Keras
Robot Operating System (ONEIROS)
PyTorch Facebook’s AI Research (FAIR) lab
OpenNN Artelnics
CNTK Microsoft Research
MXNet Apache Software Foundation (ASF)
DeeplearningKit Memkite
Deeplearning4J Eclipse Machine Learning Group
Darknet Joseph Chet Redmon
PlaidML Vertex AI, acquired by Intel
 Thank You

Department of Computer Science and Engineering