Neural Network

RNN
A recurrent neural network (RNN) is a type of artificial neural network which uses
sequential data or time series data.
These deep learning algorithms are commonly used for ordinal or temporal problems,
such as language translation, natural language processing (nlp), speech recognition, and
image captioning; they are incorporated into popular applications such as Siri, voice
search, and Google Translate.
Like feedforward and convolutional neural networks (CNNs), recurrent neural
networks utilize training data to learn.
what makes RNN so powerful is the fact that it doesn't take into consideration just the
actual input but also the previous input which allows it to memorize what happens
previously
RNN
Naïve Bayes for example it takes a sentence as a set of independent words and
precisely the frequency of each word without worrying about the composition of words
or the order of words in a sentence which makes a huge difference to form the meaning
of a sentence.
 RNN unlike those classic algorithms, works well on sequence data because it takes the
word i as input and combine with the output of word i-1, the same thing would be
applied for word i+1 and
 this is the reason it’s called recurrent neural network because clearly the neural
network apply the same operations on each word i of the sentence.
Feed Forward NN
RNN
In a feed-forward neural network, the information only moves in one direction — from
the input layer, through the hidden layers, to the output layer. The information moves
straight through the network.
Feed-forward neural networks have no memory of the input they receive and are bad at
predicting what’s coming next. Because a feed-forward network only considers the
current input, it has no notion of order in time. It simply can’t remember anything about
what happened in the past except its training.
In a RNN the information cycles through a loop. When it makes a decision, it
considers the current input and also what it has learned from the inputs it received
previously.
RNN
RNN
Back propagation is like going back in time to compute derivative of the loss function
with respect to parameters
(RNN) is a special type of artificial neural network adapted to work for time series data
or data that involves sequences. Ordinary feedforward neural networks are only meant
for data points that are independent of each other.
RNNs are better suited to analyzing temporal, sequential data, such as text or videos.
A CNN has a different architecture from an RNN. CNNs are "feed-forward neural
networks" that use filters and pooling layers, whereas RNNs feed results back into the
network
The main and most important feature of RNN is Hidden state, which remembers some
information about a sequence. RNN have a “memory” which remembers all information
about what has been calculated.
RNN
RNN
In the examples above, for each instance of training (sentence) we map each word with
an output, if the word is name(john, Ellen …) we map it to 1. Otherwise, we map it to 0.
So to train RNN on sentences to recognize names within, the RNN architecture would be
something like that
RNN
for this training example, we have 5 words which means 5 steps so for each
step t we calculate a, y using the shared weights Wa,Wx, Wy, ba, by :
RNN
Back propagation is like going back in time to compute derivative of the loss function
with respect to parameters
After multiple iterations using several training examples, we’d be able to minimize the
loss function and the predicted output would converge to the real output. Thus, we’ll use
the optimized weights to detect names through future sentences.
Convolution Neural Network
The CNN is another type of neural network that can uncover key information in both
time series and image data. For this reason, it is highly valuable for image-related tasks,
such as image recognition, object classification and pattern recognition.
To identify patterns within an image, a CNN leverages principles from linear algebra,
such as matrix multiplication. CNNs can also classify audio and signal data.
CNN
A Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm that
can take in an input image, assign importance (learnable weights and biases) to various
aspects/objects in the image, and be able to differentiate one from the other.
 The pre-processing required in a ConvNet is much lower as compared to other
classification algorithms.
While in primitive methods filters are hand-engineered, with enough training,
ConvNets have the ability to learn these filters/characteristics.
Why CNN??
Regular artificial neural networks do not scale very well. For example, in CIFAR, a
dataset that is commonly used for training computer vision models, the images are only
of size 32x32 px and have 3 color channels. That means that a single fully-connected
neuron in a first hidden layer of this neural network would have 32x32x3 = 3072
weights. It is still manageable. But now imagine a bigger image, for example,
300x300x3. It would have 270,000 weights (training of which demands so much
computational power)!
A huge neural network like that demands a lot of resources but even then remains
prone to overfitting because the large number of parameters enable it to just memorize
the dataset.
CNN
Convolution is a mathematical operation that allows the merging of two sets of
information. In the case of CNN, convolution is applied to the input data to filter the
information and produce a feature map.
CNNs use parameter sharing. All neurons in a particular feature map share weights
which makes the whole system less computationally intense.
CNN ,RNN, MLP
CNN
Convolution is one of the main building blocks of a CNN. The term convolution refers
to the mathematical combination of two functions to produce a third function. It merges
two sets of information.
In the case of a CNN, the convolution is performed on the input data with the use of
a filter or kernel (these terms are used interchangeably) to then produce a feature map.
We execute a convolution by sliding the filter over the input. At every location, a
matrix multiplication is performed and sums the result onto the feature map.
CNN
CNN
CNN
We preform numerous convolutions on our input, where each operation uses a different
filter. This results in different feature maps. In the end, we take all of these feature maps
and put them together as the final output of the convolution layer.
Just like any other Neural Network, we use an activation function to make our output
non-linear. In the case of a Convolutional Neural Network, the output of the convolution
will be passed through the activation function. This could be the ReLU activation
function.
CNN
Max Pooling example
Average Pooling
Output of Pooling
Because the size of the feature map is always smaller than the input, we have to do something to
prevent our feature map from shrinking. This is where we use padding.
A layer of zero-value pixels is added to surround the input with zeros, so that our feature map
will not shrink. In addition to keeping the spatial size constant after performing convolution,
padding also improves performance and makes sure the kernel and stride size will fit in the input.
After a convolution layer, it is common to add a pooling layer in between CNN layers. The
function of pooling is to continuously reduce the dimensionality to reduce the number of
parameters and computation in the network. This shortens the training time and
controls overfitting.
The most frequent type of pooling is max pooling, which takes the maximum value in each
window. These window sizes need to be specified beforehand. This decreases the feature map size
while at the same time keeping the significant information.
CNN
Padding. Padding expands the input matrix by adding fake pixels to the borders of the
matrix. This is done because convolution reduces the size of the matrix. For example, a
5x5 matrix turns into a 3x3 matrix when a filter goes over it.
Striding. It often happens that when working with a convolutional layer, you need to
get an output that is smaller than the input. One way to achieve this is to use a pooling
layer. Another way to achieve this is to use striding. The idea behind stride is to skip
some areas when the kernel slides over: for example, skipping every 2 or 3 pixels. It
reduces spatial resolution and makes the network more computationally efficient.
CNN
Thus when using a CNN, the four important hyperparameters we have to decide on
are:
the kernel size
the filter count (that is, how many filters do we want to use)
CNN vs RNN
CNN RNN
It is suitable for spatial data such as images. RNN is suitable for temporal data, also called
sequential data.
CNN is considered to be more powerful than RNN includes less feature compatibility when
RNN. compared to CNN.
This network takes fixed size inputs and RNN can handle arbitrary input/output lengths.
generates fixed size outputs.
CNN is a type of feed-forward artificial neural RNN unlike feed forward neural networks - can
network with variations of multilayer perceptrons use their internal memory to process arbitrary
designed to use minimal amounts of sequences of inputs.
preprocessing.
CNNs use connectivity pattern between the Recurrent neural networks use time-series
neurons. This is inspired by the organization of information - what a user spoke last will impact
the animal visual cortex, whose individual what he/she will speak next.
neurons are arranged in such a way that they
respond to overlapping regions tiling the visual
field.
CNNs are ideal for images and video processing. RNNs are ideal for text and speech analysis.
Example
Examples of CNN in computer vision are face recognition, image classification etc.

Applications of Recurrent Neural Networks: