DEEP LEARNING

UNIT-IV
AUTO ENCODERS
What are Autoencoders?
 An autoencoder neural network is an
Unsupervised Machine learning algorithm that is designed
to receive an input and transform it into a different
representation. Autoencoders can be used to compress the
data and reduce its dimensionality.
 Autoencoders are a specific type of feedforward neural
networks where the input is the same as the output. They
compress the input into a lower-dimensional code and then
reconstruct the output from this representation. The code is a
compact “summary” or “compression” of the input, also
Architecture
An Autoencoder consists of three layers:
1. Encoder : It compresses the input image into a latent space
representation. It encodes the input image as a compressed
representation in a reduced dimension. The compressed image
is a distorted version of the original image.
2. Code : The Code layer represents the compressed input fed to
the decoder layer.
3. Decoder : This layer decodes the encoded image back to the
original dimension. The decoded image is a lossy
reconstruction of the original image and it is reconstructed from
the latent space representation.
 The layer between the encoder and decoder, ie. the code is also
Architecture
 The encoder compresses the input and produces the
code, the decoder then reconstructs the input only
using this code.
 To build an autoencoder we need 3 things: an
encoding method, decoding method, and a loss
function to compare the output with the target.
Architecture
 Both the encoder and
decoder are fully-
connected feedforward
neural networks.
 Code is a single layer
of an ANN with the
dimensionality of our
choice. The number of
nodes in the code layer
is
a hyperparameter that Fig. Detailed visualization of an autoencoder
Architecture
 First the input passes through the encoder, which is a
fully-connected ANN, to produce the code.
 The decoder, which has the similar ANN structure,
then produces the output only using the code.
 The goal is to get an output identical with the input.
 Note that the decoder architecture is the mirror
image of the encoder. The only requirement is the
dimensionality of the input and output needs to be
the same.
Properties and Hyperparameters
Properties of Autoencoders:
1. Data-specific: Autoencoders are only able to
compress data similar to what they have been
trained on.
2. Lossy: The decompressed outputs will be degraded
compared to the original inputs.
3. Learned automatically from examples: It is easy
to train specialized instances of the algorithm that
will perform well on a specific type of input.
Properties and Hyperparameters
Hyperparameters of Autoencoders: There
are 4 hyperparameters that we need to set before training an
autoencoder:
1. Code size: It represents the number of nodes in the middle layer.
Smaller size results in more compression.
2. Number of layers: The autoencoder can consist of as many
layers as we want.
3. Number of nodes per layer: The number of nodes per layer
decreases with each subsequent layer of the encoder, and
increases back in the decoder. The decoder is symmetric to the
encoder in terms of the layer structure.
Applications of Autoencoders
Image Coloring
 Autoencoders are used for converting any black and white
picture into a colored image. Depending on what is in the picture,
it is possible to tell what the color should be.
Applications of Autoencoders
Feature variation
 It extracts only the required features of an image and generates
the output by removing any noise or unnecessary interruption.
Applications of Autoencoders
Denoising Image
 The input seen by the autoencoder is not the raw input but a
stochastically corrupted version. A denoising autoencoder is thus
trained to reconstruct the original input from the noisy version.
Applications of Autoencoders
Dimensionality Reduction
 The reconstructed image is the same as our input but with
reduced dimensions. It helps in providing the similar image with a
reduced pixel value.
Applications of Autoencoders
Watermark Removal
 It is also used for removing watermarks from images or

to remove any object while filming a video or a movie.

Types of Autoencoders
 Initially used for dimensionality reduction and feature
learning, an autoencoder concept has evolved over
the years and is now widely used for learning
generative models of data.
 Here are five popular autoencoders:

1. Denoising autoencoders
2. Sparse autoencoders
3. Deep autoencoders
4. Variational Autoencoders (for generative modelling)
1. Denoising Autoencoders
 A denoising autoencoder is a neural network model
that removes noise from corrupted or noisy data by
learning to reconstruct the original data from the noisy
version. It is trained to minimize the difference
between the original and reconstructed data.
Denoising autoencoders can be stacked to form deep
networks for improved performance.
 The architecture can be adapted to other data types,
such as images, audio, and text. Adding noise can be
customized, e.g., Salt-and-Pepper, Gaussian noise.
1. Denoising Autoencoders
 In denoising autoencoders, we feed a noisy version of
the image, where noise has been added via digital
alterations. The noisy image is fed to the encoder-
decoder architecture, and the output is compared with
the ground truth image.
1. Architecture of Denoising
Autoencoders
 The architecture of a denoising autoencoder (DAE) is
similar to that of a standard autoencoder. It consists of
an encoder, which maps the input data to a lower-
dimensional representation, or encoding, and a
decoder, which maps the encoding back to the
original data space.
Encoder:
 Encoder is a neural network with one or more hidden

layers.
1. Architecture of Denoising
Autoencoders
Decoder:
 Decoder is an expansion function that reconstructs

original data from compressed encoding.

 Input to the decoder is the encoding from an encoder,

and output is a reconstruction of the original data.

 Decoder is a neural network with one or more hidden

layers.
Architecture of Denoising Autoencoders
During training, the DAE is given a set of clean input
examples and the corresponding noisy versions of
these examples. The goal is to learn a function that
maps the noisy input to the clean output using
the encoder-decoder architecture. This is typically
done using a reconstruction loss function that
measures the difference between the clean input and
the reconstructed output. The DAE is trained to
minimize this loss by updating the weights of the
encoder and decoder through backpropagation.
 Application of DAE
Denoising autoencoders (DAEs) have many
applications in computer vision, speech processing, and
natural language processing. Some examples include:
 Image denoising:
DAEs can be used to remove noise from images, such as
Gaussian noise or salt-and-pepper noise.
 Fraud detection:
DAEs can be used to identify fraudulent transactions by
learning to reconstruct normal transactions from noisy
versions.
 Application of DAE
 Data imputation:
DAEs can impute missing values in a dataset by learning
to reconstruct the missing values from the remaining
data.
 Data compression:
DAEs can compress data by learning a compact data
representation in the encoding space.
 Anomaly detection:
DAEs can identify anomalies in a dataset by learning to
reconstruct normal data and then flagging difficult inputs
1. Denoising Autoencoders
1. Denoising Autoencoders
 2. Sparse autoencoders
 A sparse autoencoder is simply an autoencoder whose
training criterion involves a sparsity penalty.
 The idea behind Sparse Autoencoders is that we can
achieve an information bottleneck (same information
with fewer neurons) without reducing the number of
neurons in the hidden layers. The number of neurons in the
hidden layer can be greater than the number in the input
layer.
 According to the sparsity constraint, only some percentage
of nodes can be active in a hidden layer. The neurons with
 2. Sparse autoencoders
 Regularization in sparse autoencoders is a
technique used to control the learning process
and encourage the network to produce sparse
representations in the hidden layers.
 The idea is to introduce additional constraints or
penalties during training that guide the autoencoder
to use only a subset of its neurons for each input,
resulting in a sparse activation pattern. This helps in
learning more meaningful and informative features
while preventing overfitting.
 2. Sparse autoencoders
 L1 Regularization: L1 regularization adds a penalty term
to the loss function that is proportional to the absolute
values of the weights. This encourages many weights to
become exactly zero, effectively forcing some neurons to
be inactive for certain inputs and promoting sparsity.
 Kullback-Leibler (KL) Divergence Regularization: KL
divergence regularization imposes a constraint on the
activations of the hidden units to follow a specific target
distribution, often a Bernoulli distribution. This encourages
the activation of neurons to be either close to zero or one,
leading to sparsity.
2. Sparse autoencoders
2. Sparse autoencoders
Applications:
1. Dimensionality Reduction: Sparse autoencoders can be used
to perform dimensionality reduction on high-dimensional data.
By learning a compact representation of the input data in the
hidden layers, they can help reduce the amount of data while
preserving important features.
2. Data Compression: Sparse autoencoders can be used for
data compression, where the network learns a compact
representation of the input data. This compressed
representation can be stored or transmitted more efficiently
than the original data
2. Sparse autoencoders
Applications:
3. Collaborative Filtering: In recommendation systems, sparse
autoencoders can learn representations of users and items
based on their interactions in a user-item matrix. These learned
representations can then be used to provide personalized
recommendations.
4. Document Representation: For natural language processing
tasks, sparse autoencoders can be used to learn meaningful
representations of text documents. These representations can
capture semantic information and assist in tasks like document
classification and clustering.
 3. Deep Autoencoders
 A deep autoencoder extends the concept of a simple
autoencoder by adding more layers, making it capable of
learning more complex and abstract representations of
input data.
 It is composed of two, symmetrical deep-belief networks-
1. First four or five shallow layers representing the encoding
half of the net.
2. The second set of four or five layers that make up the
decoding half.
3. Deep Autoencoders

Use cases of Deep

Autoencoders
1. Image Search
2. Data Compression
3. Topic Modeling &
Information Retrieval (IR)
3. Deep Autoencoders
 Representation Learning in Natural Language
Processing: Deep autoencoders can be applied to learn
representations of text data, which can then be used for tasks
like sentiment analysis, machine translation, and text
classification.
 Drug Discovery: Deep autoencoders can help analyze
molecular data, predict chemical properties, and assist in drug
discovery by learning meaningful molecular representations
 Image Search
 Data Compression
 3. Deep Autoencoders
Encoder: The encoder part of the deep autoencoder takes the input
data and maps it to a series of hidden layers, gradually reducing the
dimensionality of the data. Each hidden layer learns to capture
higher-level features and abstractions.
Bottleneck Layer: The bottleneck layer is the middle layer of the
network where the data is represented in a lower-dimensional space.
This layer captures the most important features of the input data.
Decoder: The decoder part of the deep autoencoder takes the
bottleneck representation and maps it back to the original input
4. Variational Autoencoders
 A Variational Autoencoder (VAE) is a deep learning
model that can generate new data samples. It
comprises two parts: an encoder network and
a decoder network. The encoder network maps the
input data to a lower-dimensional latent space, and
the decoder network maps the latent representation
back to the original data space.
 Variational Autoencoders (VAEs) are generative
models that learn a dataset's underlying probability
distribution and generate new samples.
4. Variational Autoencoders
 They use an encoder-decoder architecture,
where the encoder maps the input data to a
latent representation, and the decoder tries to
reconstruct the original data from this latent
representation. The VAE is trained to minimize
the difference between the original data and the
reconstructed data, allowing it to learn the
underlying distribution of the data and generate
new samples that follow that same distribution.
4. Variational Autoencoders
 One of the key advantages of VAEs is that they can
generate new data samples similar to the training
data. This is because the latent space learned by the
VAE is continuous, which allows the decoder to
generate new data points that are smoothly
interpolated between the training data points.
 VAEs have some applications, including image
generation, text generation, and density estimation.
They have also been used in various fields,
including computer vision, natural language
4. Variational Autoencoders
For example :
 This is what a variational autoencoder would learn

from the input:

Applications
 Data Generation and Synthesis: VAEs can generate new data
samples that resemble the training data. This has applications in
creating realistic images, text, music, and other types of creative
content.
 Image-to-Image Translation: VAEs can be used for tasks like
image style transfer, where they learn the latent representations
of images and then manipulate these representations to generate
images in different styles or domains.
 Data Augmentation: VAEs can generate augmented versions of
existing data, aiding in improving the performance of machine
learning models by increasing the diversity of the training dataset.
 Applications
 Natural Language Generation: In natural language processing,
VAEs can generate coherent and contextually relevant text, which is
useful for chatbots, content generation, and creative writing.
 Style Transfer and Image Manipulation: VAEs can be used to
separate content and style information from images, allowing users
to manipulate and transfer artistic styles between images.
 Image Inpainting: VAEs can fill in missing parts of an image, which
has applications in image editing, restoration, and completion.
 Super-Resolution Imaging: VAEs can be used to enhance the
resolution of images by learning a mapping between low-resolution
and high-resolution images.
Generative Adversarial Networks(GANs)
 Generative Adversarial Networks or GANs are types
of Neural Networks used for Unsupervised learning
 GANs were introduced in 2014 by Ian J. Goodfellow
and teammates.
 Generative which means that there is a network
that is constantly generating new data and the
second word is Adversarial which means that it
involves two networks opposing each other and
Network simply means an order of data that is
Generative Adversarial Networks(GANs)
Generative Adversarial Networks(GANs)
 It consists of 2 models that automatically discover and
learn the patterns in input data.
 GANs consists of two networks, a Generator G(x), and a
Discriminator D(x). They both play an adversarial game
where the generator tries to fool the discriminator by
generating data similar to those in the training set. The
Discriminator tries not to be fooled by identifying fake
data from real data. They both work simultaneously to
learn and train complex data like audio, video or image
files.
GAN
 Shown below is an example of a GAN. There is a database that has real
100 rupee notes. The generator neural network generates fake 100 rupee
notes. The discriminator network will help identify the real and fake notes.
 What is a Generator?
 A Generator in GANs is a neural network that creates fake data to be
trained on the discriminator. It learns to generate plausible data. The
generated examples/instances become negative training examples for the
discriminator. It takes a fixed-length random vector carrying noise as input
and generates a sample.
 The main aim of the Generator is to make the discriminator classify its
output as real.
What is a Generator?
The part of the GAN that trains the Generator includes:
 noisy input vector

 generator network, which transforms the random

input into a data instance

 discriminator network, which classifies the

generated data
 generator loss, which penalizes the Generator for

failing to dolt the discriminator

 What is a Generator?
 The backpropagation method is used to adjust each weight in
the right direction by calculating the weight's impact on the
output. It is also used to obtain gradients and these gradients
can help change the generator weights.
 What is a Discriminator?
 The Discriminator is a neural network that identifies
real data from the fake data created by the Generator.
The discriminator's training data comes from different
two sources:
 The real data instances, such as real pictures of birds,
humans, currency notes, etc., are used by the
Discriminator as positive samples during training.
 The fake data instances created by the Generator are
used as negative examples during the training process.
What is a Discriminator?
 What is a Discriminator?
 In the process of training the discriminator, the discriminator
classifies both real data and fake data from the generator. The
discriminator loss penalizes the discriminator for misclassifying a
real data instance as fake or a fake data instance as real.
 The discriminator updates its weights through backpropagation
from the discriminator loss through the discriminator network.
What is a Discriminator?
Application of GANs
 With the help of DCGANs, you can train
images of cartoon characters for
generating faces of anime characters as
well as Pokemon characters.
 Application of GANs
 GANs can be trained on the images of humans to
generate realistic faces. The faces that you see below
have been generated using GANs and do not exist in
reality.
Application of GANs
 GANs can build realistic images from textual descriptions of
objects like birds, humans, and other animals. We input a
sentence and generate multiple images fitting the
description.
Steps for Training GAN
 Define the problem
 Choose the architecture of GAN
 Train discriminator on real data
 Generate fake inputs for the generator
 Train discriminator on fake data
 Train generator with the output of the
discriminator
Types of GANs.
 Vanilla GANs
 Deep Convolutional GANs (DCGANs)
 Conditional GANs
 Super Resolution GANs