Neural Networks And Their
Statistical Application
By Clint Hagen
Statistics Senior Seminar 2006
Outline
What Neural Networks are and why they
are desirable
How the process works and appropriate
statistical applications
Basic Architectures and algorithms
Applications
Drawbacks and limitations
Demonstration using “NeuroShell 2”
The original analyst
What are they?
Computer algorithms designed to mimic
human brain function
Set of simple computational units which
are highly interconnected
Human Brain Function
Neural Network Function
Some Similarities
Why Neural Networks are desirable
Human brain can generalize from abstract
Recognize patterns in the presence of
noise
Recall memories
Make decisions for current problems
based on prior experience
Why Desirable in Statistics
Prediction of future events based on past
experience
Able to classify to nearest pattern in
memory, doesn’t have to be exact
Predict latent variables that are not easily
measured
Non-linear regression problems
What are Neural Networks?
The computational ability of a
digital computer combined with
the desirable functions of the
human brain.
How the Process Works
Terminology, when to use neural
networks and why they are used
in statistical applications.
Terminology
Input: Explanatory variables also referred
to as “predictors”.
Neuron: Individual units in the hidden
layer(s) of a neural network.
Output: Response variables also called
“predictions”.
Hidden Layers: Layers between input and
output that an apply activation function.
Terminology
Weights: Result (parameters) of an
objective function (usually sum of squares
error) used while training a network.
Backpropagation: Most popular training
method for neural networks.
Network training: To find values of network
parameters (weights) for performing a
particular task.
Terminology
Patterns: Set of predictors with their actual
output used in training the network
When to use neural networks
Use for huge data sets (i.e. 50 predictors
and 15,000 observations) with unknown
distributions
Smaller data sets with outliers as neural
networks are very resistant to outliers
Why Neural Networks in Statistics?
The methodology is seen as a new
paradigm for data analysis where models
are not explicitly stated but rather implicitly
defined by the network.
Advanced pattern recognition capabilities
Allows for analysis where traditional
methods might be extremely tedious or
nearly impossible to interpret.
Basic Architectures
Feed Forward
Feed-forward method trained using
backpropagation (backpropagation
network) is used in time series prediction
problems most often. It is the most
commonly used algorithm.
We will see this algorithm in more detail
soon
Adaline Network
Pattern recognition network
Essentially a single layer backpropagation
network
Only recognizes exact training patterns
Hopfield Model
The Hopfield model is used as an auto-
associative memory to store and recall a
set of bitmap images.
Associative recall of images, given
incomplete or corrupted version of a
stored image the network can recall the
original
Boltzmann Machine
The Boltzmann machine is a stochastic
version of the Hopfield model.
Used for optimization problems such as
the classic traveling salesman problem
Note
Those are only a few of the more
common network structures.
Advanced users can build
networks designed for a particular
problem in many software
packages readily available on the
market today.
Feed Forward Network Trained
Using Backpropagation
Structure
Input Input Input Input
One-way only
Can have multiple
hidden layers
Each layer can have Hidden Hidden Hidden Hidden
Layer Layer Layer Layer
independent number
of neurons
Each layer fully
connected to the next
layer. Output Output Output Output
Feed-Forward Design
Alternate Structure
Predictor 1 Predictor 2 Predictor 3
Neuron i Neuron j
Wik
Wjl
Neuron k Neuron l
Output t
Weights
Each connection (arrow) in the previous
diagram has a weight, also called the
synaptic weight
The function of these weights is to reduce
error between desired output and actual
output
Weights
Weights are adjustable
Weight Wij is interpreted as the strength of
the connection between the jth unit and the
ith unit
Weights are computed in opposite
direction as the networks runs
Netinput ij = ∑ wij * outputj + µi
µi is a threshold for neuron i
Threshold
Each neuron takes its net input and
applies an activation function to it
The output of the jth neuron (activation
value) is g(∑ wij * xi) where g(·) is the
activation function and xi is the output of
the ith unit connected to j
If the net input exceeds the threshold the
neuron will “fire”
Activation Function
The only practical requirement for an
activation function is that it be
differentiable
Sigmoid function is commonly used
g(netinput) = 1/(1+ exp-(netinput))
Or a simple binary threshold unit
Ө(netinput) = {1 ,if netinput ≥ 0 ; 0 ,
otherwise}
Backpropagation
The backpropagation algorithm is a
method to find weights for a multilayered
feed forward network.
It has been shown that a feed forward
network trained using backpropagation
with sufficient number of units can
approximate any continuous function to
any level of accuracy
Training the Network
Neural Networks must be first trained
before being used to analyze new data
Process entails running patterns through
the network until the network has “learned”
the model to apply to future data
Can take a long time for noisy data
Usually doesn’t converge with desired
output, but an acceptable value close to
desired can be achieved
New Data
Once the network is trained new data can
be run through it
The network will classify new data based
on the previous data it trained with
If an exact match can not be found it will
match with the closest found in memory
Regression and Neural Networks
Objective of regression problem is to find
coefficients that minimize sum of errors
To find coefficients we must have a
dataset that includes the independent
variable and associated values of the
dependent variable. (very similar to
training the network)
Equivalent to a single layer feed forward
network
Regression
Independent variables correspond to
predictors
Coefficients β correspond to weights
The activation function is the identity
function
To find weights in a neural network we use
backpropagation and a cost function
Difference in Neural Networks
The difference in the two approaches is
that multiple linear regression has a closed
form solution for the coefficients, while
neural networks use an iterative process.
In regression models a functional form is
imposed on the data
In the case of multiple linear regression
this assumption is that the outcome is
related a linear combination of the
independent variables.
If this assumption is not correct, it will lead
to error in the prediction
An alternate approach is not to assume
any functional relationship between the
independent variables (predictors) and let
the data define the functional form.
This is the basis of the power of the neural
networks
This is very useful when you have no idea
of the functional relationship between the
dependent and independent variables
If you had an idea, you’d be better off
using a regression model
Drawbacks and Limitations
Neural Networks can be extremely hard to
use
The programs are filled with settings you
must input and a small error will cause
your predictions to have error also
The results can be very hard to interpret
as well
Drawbacks and Limitations
Neural networks should not be used when
traditional methods are appropriate
Since they are data dependent
performance will improve as sample size
increases
Regression performs better when theory
or experience indicates an underlying
relationship
A short demonstration using
“NeuroShell 2”
