Mohammed Bennamoun, profile picture
Uploaded byMohammed Bennamoun
12,766 views

Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation

The document provides information about multi-layer perceptrons (MLPs) and backpropagation. It begins with definitions of perceptrons and MLP architecture. It then describes backpropagation, including the backpropagation training algorithm and cycle. Examples are provided, such as using an MLP to solve the exclusive OR (XOR) problem. Applications of backpropagation neural networks and options like momentum, batch vs sequential training, and adaptive learning rates are also discussed.

Embed presentation

Downloaded 799 times
1 CS407 Neural Computation Lecture 5: The Multi-Layer Perceptron (MLP) and Backpropagation Lecturer: A/Prof. M. Bennamoun
2 What is a perceptron and what is a Multi-Layer Perceptron (MLP)?
3 What is a perceptron? wk1 x1 wk2 x2 wkm xm ... ... Σ Bias bk ϕ(.) vk Input signal Synaptic weights Summing junction Activation function Output yk bxwv kj m j kjk += ∑=1 )(vy kk ϕ= )()( ⋅=⋅ signϕ Discrete Perceptron: shapeS −=⋅)(ϕ Continous Perceptron:
4 Activation Function of a perceptron vi +1 -1 Signum Function (sign) )()( ⋅=⋅ signϕ Discrete Perceptron: shapesv −=)(ϕ Continous Perceptron: vi +1
5 MLP Architecture The Multi-Layer-Perceptron was first introduced by M. Minsky and S. Papert in 1969 Type: Feedforward Neuron layers: 1 input layer 1 or more hidden layers 1 output layer Learning Method: Supervised
6 Terminology/Conventions Arrows indicate the direction of data flow. The first layer, termed input layer, just contains the input vector and does not perform any computations. The second layer, termed hidden layer, receives input from the input layer and sends its output to the output layer. After applying their activation function, the neurons in the output layer contain the output vector.
7 Why the MLP? The single-layer perceptron classifiers discussed previously can only deal with linearly separable sets of patterns. The multilayer networks to be introduced here are the most widespread neural network architecture – Made useful until the 1980s, because of lack of efficient training algorithms (McClelland and Rumelhart 1986) – The introduction of the backpropagation training algorithm.
8 Different Non-Linearly Separable Problems http://www.zsolutions.com/light.htm Structure Types of Decision Regions Exclusive-OR Problem Classes with Meshed regions Most General Region Shapes Single-Layer Two-Layer Three-Layer Half Plane Bounded By Hyperplane Convex Open Or Closed Regions Arbitrary (Complexity Limited by No. of Nodes) A AB B A AB B A AB B B A B A B A
9 What is backpropagation Training and how does it work?
10 Supervised Error Back-propagation Training – The mechanism of backward error transmission (delta learning rule) is used to modify the synaptic weights of the internal (hidden) and output layers • The mapping error can be propagated into hidden layers – Can implement arbitrary complex/output mappings or decision surfaces to separate pattern classes • For which, the explicit derivation of mappings and discovery of relationships is almost impossible – Produce surprising results and generalizations What is Backpropagation?
11 Architecture: Backpropagation Network The Backpropagation Net was first introduced by G.E. Hinton, E. Rumelhart and R.J. Williams in 1986 Type: Feedforward Neuron layers: 1 input layer 1 or more hidden layers 1 output layer Learning Method: Supervised Reference: Clara Boyd
12 Backpropagation Preparation Training Set A collection of input-output patterns that are used to train the network Testing Set A collection of input-output patterns that are used to assess network performance Learning Rate-α A scalar parameter, analogous to step size in numerical integration, used to set the rate of adjustments
13 Backpropagation training cycle 1/ Feedforward of the input training pattern 2/ Backpropagation of the associated error3/ Adjustement of the weights Reference Eric Plammer
14 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
15 Source: Fausett, L., Fundamentals of Neural Networks, Prentice Hall, 1994. Notation -- p. 292 of FausettNotation -- p. 292 of Fausett BP NN With Single Hidden Layer kjw , jiv , I/P layer O/P layer Hidden layer Reference: Dan St. Clair Fausett: Chapter 6
16 Notation x = input training vector t = Output target vector. δk = portion of error correction weight for wjk that is due to an error at output unit Yk; also the information about the error at unit Yk that is propagated back to the hidden units that feed into unit Yk δj = portion of error correction weight for vjk that is due to the backpropagation of error information from the output layer to the hidden unit Zj α = learning rate. voj = bias on hidden unit j wok = bias on output unit k
17 Source: Fausett, L., Fundamentals of Neural Networks, Prentice Hall, 1994. Hyberbolic tangent Binary step Activation Functions )](1[*)()( )exp(1 1 )( ' xfxfxf x xf −= −+ = Should be continuos, differentiable, and monotonically non-decreasing. Plus, its derivative should be easy to compute.
18 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
19 Fausett, L., pp. 294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
20 Fausett, L., pp. 294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
21 Fausett, L., pp. 294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
22 Fausett, L., pp. 294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
23 Let’s examine Training Algorithm Equations [ ]nxxX ...1= [ ]pvvV ,01,00 ...= Y1 Z1 Z2 Z3 X1 X2 X3 1 1 v2,1 Vectors & matrices make computation easier.           = pnn p vv vv V ,1, ,11,1 ... ......... ... [ ])_(...)_( _ 1 0 pinzfinzfZ XVVinZ = += Step 4 computation becomes [ ]mwwW ,01,00 ...=           = mpp m ww ww W ,1, ,11,1 ... ......... ... Step 5 computation becomes [ ])_(...)_( _ 1 0 minyfinyfY ZWWinY = +=
24 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
25 Generalisation Once trained, weights are held contstant, and input patterns are applied in feedforward mode. - Commonly called “recall mode”. We wish network to “generalize”, i.e. to make sensible choices about input vectors which are not in the training set Commonly we check generalization of a network by dividing known patterns into a training set, used to adjust weights, and a test set, used to evaluate performance of trained network
26 Generalisation … Generalisation can be improved by – Using a smaller number of hidden units (network must learn the rule, not just the examples) – Not overtraining (occasionally check that error on test set is not increasing) – Ensuring training set includes a good mixture of examples No good rule for deciding upon good network size (# of layers, # units per layer) Usually use one input/output per class rather than a continuous variable or binary encoding
27 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
28 Example 1 The XOR function could not be solved by a single layer perceptron network The function is: X Y F 0 0 0 0 1 1 1 0 1 1 1 0 Reference: R. Spillman
29 XOR Architecture x y fv11 Σ v01 v21 fv12 Σ v02 v22 fw11 Σ w01 w21 1 1 1
30 Initial Weights Randomly assign small weight values: x y f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1
31 Feedfoward – 1st Pass x y f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1 Training Case: (0 0 0) 0 0 1 1 zin1 = -.3(1) + .21(0) + .15(0) = -.3 Activation function f: z1 = .43 zin2 = .25(1) -.4(0) + .1(0) z2 = .56 1 yin1 = -.4(1) - .2(.43) +.3(.56) = -.318 y1 = .42 (not 0) x e xf − + = 1 1 )(
32 Backpropagate 0 0 f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1 δ1 = (t1 – y1)f’(y_in1) =(t1 – y1)f(y_in1)[1- f(y_in1)] δ1 = (0 – .42).42[1-.42] = -.102 δ_in1 = δ1w11 = -.102(-.2) = .02 δ1 = δ_in1f’(z_in1) = .02(.43)(1-.43) = .005 δ_in2 = δ1w21 = -.102(.3) = -.03 δ2 = δ_in2f’(z_in2) = -.03(.56)(1-.56) = -.007
33 Calculate the Weights – First Pass 0 0 f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1 10 11 2,1 αδ αδ =∆ ==∆ w jzw jj jj ijij v jxv αδ αδ =∆ ==∆ 0 2,1 102.0 −=∆w 0571.)56)(.102.(2121 −=−==∆ zw δ 0439.)43)(.102.(1111 −=−==∆ zw δ 005.01 −=∆v 007.02 −=∆v 0)0)(005(.1111 ===∆ xv δ 0)0)(007.(1212 =−==∆ xv δ 0)0)(005(.2121 ===∆ xv δ 0)0)(007.(2222 =−==∆ xv δ
34 Update the Weights – First Pass 0 0 f.21 Σ -.305 .15 f-.4 Σ .243 .1 f-.044Σ -.502 .243 1 1 1
35 Final Result After about 500 iterations: x y f1 Σ -1.5 1 f1 Σ -.5 1 f-2 Σ -.5 1 1 1 1
36 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
37 Example 2 [ ]08.06.0=X          − = 2 1 1 w [ ]10 −=w[ ]1000 −=v           = 130 221 012 v 9.0=t Y1 Z1 Z2 Z3 X1 X2 X3 1 1 v2,1 α = 0.3 Desired output for X input )1( 1 )( x e xf − + = m = 1 p = 3 n = 3 Reference: Vamsi Pegatraju and Aparna Patsa
38 Primary Values: Inputs to Epoch - I X=[0.6 0.8 0]; W=[-1 1 2]’; W0=[-1]; V= 2 1 0 1 2 2 0 3 1 V0=[ 0 0 -1]; Target t=0.9; α = 0.3;
39 Epoch – I Step 4: Z_in= V0+XV = [ 2 2.2 0.6]; Z=f([Z_in])=[ 0.8808 0.9002 0.646]; Step 5: Y_in = W0+ZW = [0.3114]; Y=f([Z_in])=0.5772; Sum of Squares Error obtained originally: (0.9 – 0.5772)2 = 0.1042
40 Step 6: Error = tk – Yk = 0.9 – 0.5772 Now we have only one output and hence the value of k=1. δ1= (t1 – y1 )f’(Y_in1) We know f’(x) for sigmoid = f(x)(1-f(x)) ⇒ δ1 = (0.9 −0.5772)(0.5772)(1−0.5772) = 0.0788
41 For intermediate weights we have (j=1,2,3) ∆Wj,k=α δκΖj = α δ1Ζj ⇒ ∆W1=(0.3)(0.0788)[0.8808 0.9002 0.646]’ =[0.0208 0.0213 0.0153]’; Bias ∆W0,1=α δ1= (0.3)(0.0788)=0.0236;
42 Step 7: Backpropagation to the first hidden layer For Zj (j=1,2,3), we have δ_inj = ∑k=1..m δκWj,k= δ1Wj,1 ⇒ δ_in1=-0.0788;δ_in2=0.0788;δ_in3=0.1576; δj= δ_injf’(Z_inj) => δ1=-0.0083; δ2=0.0071; δ3=0.0361;
43 ∆Vi,j = αδjXi ⇒ ∆V1 = [-0.0015 -0.0020 0]’; ⇒ ∆V2 = [0.0013 0.0017 0]’; ⇒ ∆V3 = [0.0065 0.0087 0]’; ∆V0=α[δ1 δ2 δ3] = [ -0.0025 0.0021 0.0108]; X=[0.6 0.8 0]
44 Step 8: Updating of W1, V1, W0, V0 Wnew= Wold+∆W1=[-0.9792 1.0213 2.0153]’; Vnew= Vold+∆V1 =[1.9985 1.0013 0.065;0.998 2.0017 2.0087; 0 3 1]; W0new = -0.9764; V0new = [-0.0025 0.0021 -0.9892]; Completion of the first epoch.
45 Primary Values: Inputs to Epoch - 2 X=[0.6 0.8 0]; W=[-0.9792 1.0213 2.0153]’; W0=[-0.9792]; V=[1.9985 1.0013 0.065; 0.998 2.0017 2.0087; 0 3 1]; V0=[ -0.0025 0.0021 -0.9892]; Target t=0.9; α = 0.3;
46 Epoch – 2 Step 4: Z_in=V0+XV=[1.995 2.2042 0.6217]; Z=f([Z_in])=[ 0.8803 0.9006 0.6506]; Step 5: Y_in = W0+ZW = [0.3925]; Y=f([Z_in])=0.5969; Sum of Squares Error obtained from first epoch: (0.9 – 0.5969)2 = 0.0918
47 Step 6: Error = tk – Yk = 0.9 – 0.5969 Now again, as we have only one output, the value of k=1. δ1= (t1 – y1 )f’(Y_in1) =>δ1 = (0.9 −0.5969)(0.5969)(1−0.5969) = 0.0729
48 For intermediate weights we have (j=1,2,3) ∆Wj,k=α δκΖj = α δ1Ζj ⇒ ∆W1=(0.3)*(0.0729)* [0.8803 0.9006 0.6506]’ =[0.0173 0.0197 0.0142]’; Bias ∆W0,1=α δ1= 0.0219;
49 Step 7: Backpropagation to the first hidden layer For Zj (j=1,2,3), we have δ_inj = ∑k=1..m δκWj,k= δ1Wj,1 ⇒ δ_in1=-0.0714;δ_in2=0.0745;δ_in3=0.1469; δj= δ_injf’(Z_inj) => δ1=-0.0075; δ2=0.0067; δ3=0.0334;
50 ∆Vi,j = αδjXi ⇒ ∆V1 = [-0.0013 -0.0018 0]’; ⇒ ∆V2 = [0.0012 0.0016 0]’; ⇒ ∆V3 = [0.006 0.008 0]’; ∆V0=α[δ1 δ2 δ3] = [ -0.0022 0.002 0.01];
51 Step 8: Updating of W1, V1, W0, V0 Wnew= Wold+∆W1=[-0.9599 1.041 2.0295]’; Vnew= Vold+∆V1 =[1.9972 1.0025 0.0125; 0.9962 2.0033 2.0167; 0 3 1]; W0new = -0.9545; V0new = [-0.0047 0.0041 -0.9792]; Completion of the second epoch.
52 Z_in=V0+XV=[1.9906 2.2082 0.6417]; =>Z=f([Z_in])=[ 0.8798 0.9010 0.6551]; Step 5: Y_in = W0+ZW = [0.4684]; => Y=f([Z_in])=0.6150; Sum of Squares Error at the end of the second epoch: (0.9 – 0.615)2 = 0.0812. From the last two values of Sum of Squares Error, we see that the value is gradually decreasing as the weights are getting updated.
53 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
54 Functional Approximation Multi-Layer Perceptrons can approximate any continuous function by a two-layer network with squashing activation functions. If activation functions can vary with the function, can show that a n-input, m-output function requires at most 2n+1 hidden units. See Fausett: 6.3.2 for more details.
55 Function Approximators Example: a function h(x) approximated by H(w,x)
56 Applications We look at a number of applications for backpropagation MLP’s. In each case we’ll examine –Problem to be solved –Architecture Used –Results Reference: J.Hertz, A. Krogh, R.G. Palmer, “Introduction to the Theory of Neural Computation”, Addison Wesley, 1991
57 NETtalk - Specifications Problem is to convert written text to speech. Conventionally, this is done by hand-coded linguistic rules, such as the DECtalk system. NETtalk uses a neural network to achieve similar results Input is written text Output is choice of phoneme for speech synthesiser
58 NETtalk - architecture e c ah t oT n 7 letter sliding window, generating phoneme for centre character. Input units use 1 of 29 code. => 203 input units (=29x7) 80 hidden units, fully interconnected 26 output units, 1 of 26 code representing most likely phoneme
59 NETtalk - Results 1024 Training Set After 10 epochs - intelligible speech After 50 epochs - 95% correct on training set - 78% correct on test set Note that this network must generalise - many input combinations are not in training set Results not as good as DECtalk, but significantly less effort to code up.
60 Sonar Classifier Task - distinguish between rock and metal cylinder from sonar return of bottom of bay Convert time-varying input signal to frequency domain to reduce input dimension. (This is a linear transform and could be done with a fixed weight neural network.) Used a 60-x-2 network with x from 0 to 24 Training took about 200 epochs. 60-2 classified about 80% of training set; 60-12-2 classified 100% training, 85% test set
61 ALVINN Drives 70 mph on a public highway 30x32 pixels as inputs 30 outputs for steering 30x32 weights into one out of four hidden unit 4 hidden units
62 Navigation of a Car Task is to control a car on a winding road Inputs are a 30x32 pixel image from a video camera on roof, 8x32 image from a range finder => 1216 inputs 29 hidden units 45 output units arranged in a line, 1-of-45 code representing hard-left..straight-ahead..hard-right
63 Navigation of Car - Results Training set of 1200 simulated road images Trained for 40 epochs Could drive at 5 km/hr on road, limited by calculation speed of feed-forward network. Twice as fast as best non-net solution
64 Backgammon Trained on 3000 example board scenarios of (position, dice, move) rated from -100 (very bad) to +100 (very good) from human expert. Some important information such as “pip- count” and “degree-of-trapping” was included as input. Some “noise” added to input set (scenarios with random score) Handcrafted examples added to training set to correct obvious errors
65 Backgammon results 459 inputs, 2 hidden layers, each 24 units, plus 1 output for score (All possible moves evaluated) Won 59% against a conventional backgammon program (41% without extra info, 45% without noise in training set) Won computer olympiad, 1989, but lost to human expert (Not surprising since trained by human scored examples)
66 Encoder / Image Compression Wish to encode a number of input patterns in an efficient number of bits for storage or transmission We can use an autoassociative network, i.e. an M-N-M network, where we have M inputs, and N<M hidden units, M outputs, trained with target outputs same as inputs Hidden units need to encode inputs in fewer signals in the hidden layers. Outputs from hidden layer are encoded signal
67 Encoders We can store/transmit hidden values using first half of network; decode using second half. We may need to truncate hidden unit values to fixed precision, which must be considered during training. Cottrell et al. tried 8x8 blocks (8 bits each) of images, encoded in 16 units, giving results similar to conventional approaches. Works best with similar images
68 Neural network for OCR feedforward network trained using Back- propagation A B E D C Output Layer Input Layer Hidden Layer 8 10 8 8 1010
69 Pattern Recognition Post-code (or ZIP code) recognition is a good example - hand-written characters need to be classified. One interesting network used 16x16 pixel map input of handwritten digits already found and scaled by another system. 3 hidden layers plus 1-of-10 output layer. First two hidden layers were feature detectors.
70 ZIP code classifier First layer had same feature detector connected to 5x5 blocks of input, at 2 pixel intervals => 8x8 array of same detector, each with the same weights but connected to different parts of input. Twelve such feature detector arrays. Same for second hidden layer, but 4x4 arrays connected to 5x5 blocks of first hidden layer; with 12 different features. Conventional 30 unit 3rd hidden layer
71 ZIP Code Classifier - Results Note 8x8 and 4x4 arrays of feature detectors use the same weights => many fewer weights to train. Trained on 7300 digits, tested on 2000 Error rates: 1% on training, 5% on test set If cases with no clear winner rejected (i.e. largest output not much greater than second largest output), then, with 12% rejection, error rate on test set reduced to 1%. Performance improved further by removing more weights: “optimal brain damage”.
72 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
73 Heuristics for making BP Better Training with BP is more an art than science – result of own experience Normalizing the inputs – preprocessed so that its mean value is closer to zero (see “prestd” function in matlab). – input variables should be uncorrelated • by “Principal Component Analysis” (PCA). See “prepca” and “trapca” functions in Matlab.
74 Sequential vs. Batch update “Sequential” learning means that a given input pattern is forward propagated, the error is determined and back-propagated, and the weights are updated. Then the same procedure is repeated for the next pattern. “Batch” learning means that the weights are updated only after the entire set of training patterns has been presented to the network. In other words, all patterns are forward propagated, and the error is determined and back-propagated, but the weights are only updated when all patterns have been processed. Thus, the weight update is only performed every epoch. If P = # patterns in one epoch ∑= ∆=∆ P p pw P w 1 1
75 Sequential vs. Batch update i.e.in some cases, it is advantageous to accumulate the weight correction terms for several patterns (or even an entire epoch if there are not too many patterns) and make a single weight adjustment (equal to the average of the weight correction terms) for each weight rather than updating the weights after each pattern is presented. This procedure has a “smoothing effect” (because of the use of the average) on the correction terms. In some cases, this smoothing may increase the chances of convergence to a local minimum.
76 Initial weights Initial weights – will influence whether the net reaches a global (or only a local minimum) of the error and if so, how quickly it converges. – The values for the initial weights must not be too large otherwise, the initial input signals to each hidden or output unit will be likely to fall in the region where the derivative of the sigmoid function has a very small value (f’(net)~0) : so called saturation region. – On the other hand, if the initial weights are too small, the net input to a hidden or output unit will be close to zero, which also causes extremely slow learning. – Best to set the initial weights (and biases) to random numbers between –0.5 and 0.5 (or between –1 and 1 or some other suitable interval). – The values may be +ve or –ve because the final weights after training may be of either sign also.
77 Memorization vs. generalization How long to train the net: Since the usual motivation for applying a backprop net is to achieve a balance between memorization and generalization, it is not necessarily advantageous to continue training until the error actually reaches a minimum. – Use 2 disjoint sets of data during training: 1/ a set of training patterns and 2/ a set of training- testing patterns (or validation set). – Weight adjustment are based on the training patterns; however, at intervals during training, the error is computed using the validation patterns. – As long as the error for the validation decreases, training continues. – When the error begins to increase, the net is starting to memorize the training patterns too specifically (starts to loose its ability to generalize). At this point, training is terminated.
78 Early stopping Error Training time With training set (which changes wij) With validation set (which does not change wij) Stop Here ! L. Studer, IPHE-UNIL
79 Backpropagation with momentum Backpropagation with momentum: the weight change is in a direction that is a combination of 1/ the current gradient and 2/ the previous gradient. Momentum can be added so weights tend to change more quickly if changing in the same direction for several training cycles:- ∆ wij (t+1) = α δ xi + µ . ∆ wij (t) µ is called the “momentum factor” and ranges from 0 < µ < 1. – When subsequent changes are in the same direction increase the rate (accelerated descent) – When subsequent changes are in opposite directions decrease the rate (stabilizes)
80 Backpropagation with momentum… Weight update equation Momentum )1( −tw )(tw z)( αδ+tw )1( +tw Source: Fausett, L., Fundamentals of Neural Networks, Prentice Hall, 1994, pg. 305.
81 Source: Fausett, L., Fundamentals of Neural Networks, Prentice Hall, 1994. Adaptive learning rate BP training algorithm – Adaptive Learning Rate BP training algorithm – Adaptive Learning Rate
82 Adaptive Learning rate… Adaptive Parameters: Vary the learning rate during training, accelerating learning slowly if all is well ( error, E, decreasing) , but reducing it quickly if things go unstable (E increasing). For example: Typically, a = 0.1, b = 0.5      >∆ <∆+ =+ otherwise(t) 0Eif(t).b)-(1 epochsfewlastfor0Eifa(t) 1)(t α α α α
83 Matlab BP NN Architecture A neuron with a single R-element input vector is shown below. Here the individual element inputs • are multiplied by weights • and the weighted values are fed to the summing junction. Their sum is simply Wp, the dot product of the (single row) matrix W and the vector p. The neuron has a bias b, which is summed with the weighted inputs to form the net input n. This sum, n, is the argument of the transfer function f. • This expression can, of course, be written in MATLAB code as: •n = W*p + b However, the user will seldom be writing code at this low level, for such code is already built into functions to define and simulate entire networks.
84 Matlab BP NN Architecture
85 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
86 Learning Rule Similar to Delta Rule. Our goal is to minimize the error, E, which is the difference between targets, tm , and our outputs, yk m , using a least squares error measure: E = 1/2 Σk (tk - yk)2 To find out how to change wjk and vij to reduce E, we need to find ijjk v E and w E ∂ ∂ ∂ ∂ Fausett, section 6.3, p324
87 Delta Rule Derivation Hidden-to-Output       −=−= ∑∑ k 2 kk jkjk 2 )y(t 2 1 ww E hence][5.0 ∂ ∂ ∂ ∂ k kk ytE [ ] [ ]       −=      −= ∑ 2 K JKk 2 k JKJK )(t 2 1 w t 2 1 ww E inKk yfy ∂ ∂ ∂ ∂ ∂ ∂ and)(where ∑== j jKjinKinKk wzyyfy JKinKK JK ).z(y')fy(t w E −−= ∂ ∂ JK inK JK inK w y w yf ∂ ∂ −−= ∂ ∂ −−= )( ).(y')fy(t )( )y(t w E KinKKKK JK∂ ∂ Notice the difference between the subscripts k (which corresponds to any node between hidden and output layers) and K (which represents a particular node K of interest)
88 Delta Rule Derivation Hidden-to-Output )(y')f(t:definetoconvenientisIt inKK KK y−=δ jkjinkkk zzyfyt δαα ∂ ∂ α =−=−=∆ )('][ w E wThus, jk jk jk zδα=∆ jkwsummary,In )(y')f(twith inKK KK y−=δ
89 Delta Rule Derivation: Input to Hidden IJ ink ink k kk IJ k k kk v y yfyt v y yt v ∂ ∂ −−= ∂ ∂ −−= ∑∑ )('][][ E IJ∂ ∂ ])[(' v E IJ IinJJk k k IJ J Jk k k IJ ink k k xzfw v z w v y ∑∑∑ −= ∂ ∂ −= ∂ ∂ −= δδδ ∂ ∂       −=−= ∑∑ k 2 kk IJIJ 2 )y(t 2 1 v E hence][5.0 v ytE k kk ∂ ∂ ∂ ∂ and)(where ∑== j jKjinKinKk wzyyfy )(z'f:definetoconvenientisIt inJ∑= k JkkJ wδδ Notice the difference between the subscripts j and J and i and I ij k jkkiinjij xwxzfv αδδα ∂ ∂ α ==−=∆ ∑)(' v E ij
90 Delta Rule Derivation: Input to Hidden )(z'f:where inJ∑= k JkkJ wδδ ijij xv αδ=∆summaryIn
91 Backpropagation Neural Networks Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
92 Suggested Reading. L. Fausett, “Fundamentals of Neural Networks”, Prentice-Hall, 1994, Chapter 6.
93 References: These lecture notes were based on the references of the previous slide, and the following references 1. Eric Plummer, University of Wyoming www.karlbranting.net/papers/plummer/Pres.ppt 2. Clara Boyd, Columbia Univ. N.Y comet.ctr.columbia.edu/courses/elen_e4011/2002/Artificial.ppt 3. Dan St. Clair, University of Missori-Rolla, http://web.umr.edu/~stclair/class/classfiles/cs404_fs02/Misc/CS 404_fall2001/Lectures/Lect09_102301/ 4. Vamsi Pegatraju and Aparna Patsa: web.umr.edu/~stclair/class/classfiles/cs404_fs02/ Lectures/Lect09_102902/Lect8_Homework/L8_3.ppt 5. Richard Spillman, Pacific Lutheran University: www.cs.plu.edu/courses/csce436/notes/pr_l22_nn5.ppt 6. Khurshid Ahmad and Matthew Casey Univ. Surrey, http://www.computing.surrey.ac.uk/courses/cs365/

More Related Content

PPT
Multi-Layer Perceptrons
byESCOM
 
PPT
backpropagation in neural networks
byAkash Goel
 
PDF
Neural Networks: Multilayer Perceptron
byMostafa G. M. Mostafa
 
PPTX
Multi Layer Network
byInternational Islamic University
 
PPTX
Multilayer Perceptron Neural Network MLP
byAbdullah al Mamun
 
PPTX
Feed Forward Neural Network.pptx
byNoorFathima60
 
PPT
neural networks
byInstitute of Technology Telkom
 
PPTX
03 Single layer Perception Classifier
byTamer Ahmed Farrag, PhD
 
Multi-Layer Perceptrons
byESCOM
 
backpropagation in neural networks
byAkash Goel
 
Neural Networks: Multilayer Perceptron
byMostafa G. M. Mostafa
 
Multi Layer Network
byInternational Islamic University
 
Multilayer Perceptron Neural Network MLP
byAbdullah al Mamun
 
Feed Forward Neural Network.pptx
byNoorFathima60
 
neural networks
byInstitute of Technology Telkom
 
03 Single layer Perception Classifier
byTamer Ahmed Farrag, PhD
 

What's hot

PPTX
Multilayer perceptron
byomaraldabash
 
PDF
Convolutional Neural Network Models - Deep Learning
byMohamed Loey
 
PPSX
Perceptron (neural network)
byEdutechLearners
 
PDF
Classification by back propagation, multi layered feed forward neural network...
bybihira aggrey
 
PPTX
Artifical Neural Network and its applications
bySangeeta Tiwari
 
PPT
Perceptron
byNagarajan
 
PPTX
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
PDF
Convolutional neural network
byYan Xu
 
PPTX
Convolution Neural Network (CNN)
bySuraj Aavula
 
PPTX
Deep Learning - CNN and RNN
byAshray Bhandare
 
PDF
Training Neural Networks
byDatabricks
 
PPT
Back propagation
byNagarajan
 
PDF
Introduction to Neural Networks
byDatabricks
 
PDF
Deep Feed Forward Neural Networks and Regularization
byYan Xu
 
PPTX
Feedforward neural network
bySopheaktra YONG
 
PPTX
Batch normalization presentation
byOwin Will
 
PPTX
Perceptron & Neural Networks
byNAGUR SHAREEF SHAIK
 
PDF
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
PPT
Adaptive Resonance Theory
byNaveen Kumar
 
PDF
Multi Layer Perceptron & Back Propagation
bySung-ju Kim
 
Multilayer perceptron
byomaraldabash
 
Convolutional Neural Network Models - Deep Learning
byMohamed Loey
 
Perceptron (neural network)
byEdutechLearners
 
Classification by back propagation, multi layered feed forward neural network...
bybihira aggrey
 
Artifical Neural Network and its applications
bySangeeta Tiwari
 
Perceptron
byNagarajan
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
Convolutional neural network
byYan Xu
 
Convolution Neural Network (CNN)
bySuraj Aavula
 
Deep Learning - CNN and RNN
byAshray Bhandare
 
Training Neural Networks
byDatabricks
 
Back propagation
byNagarajan
 
Introduction to Neural Networks
byDatabricks
 
Deep Feed Forward Neural Networks and Regularization
byYan Xu
 
Feedforward neural network
bySopheaktra YONG
 
Batch normalization presentation
byOwin Will
 
Perceptron & Neural Networks
byNAGUR SHAREEF SHAIK
 
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
Adaptive Resonance Theory
byNaveen Kumar
 
Multi Layer Perceptron & Back Propagation
bySung-ju Kim
 

Similar to Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation

PPT
Intro to Deep learning - Autoencoders
byAkash Goel
 
PPT
nural network ER. Abhishek k. upadhyay
byabhishek upadhyay
 
PPTX
Introduction to Perceptron and Neural Network.pptx
byPoonam60376
 
PPTX
Perceptron and Sigmoid Neurons
byShajun Nisha
 
PPT
Neural Networks in Data Mining - “An Overview”
byDr.(Mrs).Gethsiyal Augasta
 
PPT
Chapter 9. Classification Advanced Methods.ppt
bySubrata Kumer Paul
 
PDF
33.-Multi-Layer-Perceptron.pdf
bygnans Kgnanshek
 
PPT
this is a Ai topic neural network ML_Lecture_4.ppt
byry54321288
 
PPT
Lec 6-bp
byTaymoor Nazmy
 
PPT
Finger Biometric - Neural Network.ppt
byRabaaouiAmmar
 
PPTX
Machine Learning DR PRKRao-PPT UNIT-II.pptx
bycsithodsdes
 
PPT
artificial neural networks Unit-2 Perceptron.ppt
byXI089SujanaMallavara
 
PPTX
Unit ii supervised ii
byIndira Priyadarsini
 
PPTX
Classification_by_back_&propagation.pptx
bySadiaSaleem301
 
PPTX
Neural net and back propagation
byMohit Shrivastava
 
PPTX
MNN
byInternational Islamic University
 
PPTX
back propagation1_presenation_lab 6.pptx
bysomeyamohsen2
 
PPTX
Feed forward back propogation algorithm .pptx
byneelamsanjeevkumar
 
PDF
Separating Hype from Reality in Deep Learning with Sameer Farooqui
byDatabricks
 
PDF
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
byManish Saraswat
 
Intro to Deep learning - Autoencoders
byAkash Goel
 
nural network ER. Abhishek k. upadhyay
byabhishek upadhyay
 
Introduction to Perceptron and Neural Network.pptx
byPoonam60376
 
Perceptron and Sigmoid Neurons
byShajun Nisha
 
Neural Networks in Data Mining - “An Overview”
byDr.(Mrs).Gethsiyal Augasta
 
Chapter 9. Classification Advanced Methods.ppt
bySubrata Kumer Paul
 
33.-Multi-Layer-Perceptron.pdf
bygnans Kgnanshek
 
this is a Ai topic neural network ML_Lecture_4.ppt
byry54321288
 
Lec 6-bp
byTaymoor Nazmy
 
Finger Biometric - Neural Network.ppt
byRabaaouiAmmar
 
Machine Learning DR PRKRao-PPT UNIT-II.pptx
bycsithodsdes
 
artificial neural networks Unit-2 Perceptron.ppt
byXI089SujanaMallavara
 
Unit ii supervised ii
byIndira Priyadarsini
 
Classification_by_back_&propagation.pptx
bySadiaSaleem301
 
Neural net and back propagation
byMohit Shrivastava
 
MNN
byInternational Islamic University
 
back propagation1_presenation_lab 6.pptx
bysomeyamohsen2
 
Feed forward back propogation algorithm .pptx
byneelamsanjeevkumar
 
Separating Hype from Reality in Deep Learning with Sameer Farooqui
byDatabricks
 
Understanding Deep Learning & Parameter Tuning with MXnet, H2o Package in R
byManish Saraswat
 

More from Mohammed Bennamoun

PDF
Artificial Neural Networks Lect3: Neural Network Learning rules
byMohammed Bennamoun
 
PDF
Artificial Neural Network Lect4 : Single Layer Perceptron Classifiers
byMohammed Bennamoun
 
PDF
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
byMohammed Bennamoun
 
PDF
Artificial Neural Networks Lect7: Neural networks based on competition
byMohammed Bennamoun
 
PDF
Artificial Neural Networks Lect1: Introduction & neural computation
byMohammed Bennamoun
 
PDF
Artificial Neural Networks Lect2: Neurobiology & Architectures of ANNS
byMohammed Bennamoun
 
PDF
Artificial Neural Networks Lect8: Neural networks for constrained optimization
byMohammed Bennamoun
 
Artificial Neural Networks Lect3: Neural Network Learning rules
byMohammed Bennamoun
 
Artificial Neural Network Lect4 : Single Layer Perceptron Classifiers
byMohammed Bennamoun
 
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
byMohammed Bennamoun
 
Artificial Neural Networks Lect7: Neural networks based on competition
byMohammed Bennamoun
 
Artificial Neural Networks Lect1: Introduction & neural computation
byMohammed Bennamoun
 
Artificial Neural Networks Lect2: Neurobiology & Architectures of ANNS
byMohammed Bennamoun
 
Artificial Neural Networks Lect8: Neural networks for constrained optimization
byMohammed Bennamoun
 

Recently uploaded

PPTX
19ECX25-CAD for VLSI Circuits Unit-1.pptx
bytamil arasan
 
PPTX
Factory_Automation used in Various Factories_Unit2_Detailed.pptx
bybekhem2008
 
PPTX
Unit 1_Introduction to Automation & Robotics.pptx
bybekhem2008
 
PDF
2025 MediaEval Medico Organizer Presentation
bySushant Gautam
 
PDF
Operating System Concepts-Linux and Shell Programming by K. Adi
byAdiseshaK
 
PPTX
First law of thermodynamics for chemical engineering.pptx
byAnonymousEBReO8v
 
DOCX
19EC704 Optical and Microwave Lab Manual.docx
bytamil arasan
 
PPTX
EcoSath Hackathon Final Presentation Based on ESG Data
byshashirwr2016
 
DOCX
23AE205-IMAGE AND VLSI SIGNAL PROCESSING LAB.docx
bytamil arasan
 
PDF
From Orchestration to Platform The Missing Puzzles to Unify Polyglot Services...
byRichárd Kovács
 
PPTX
Pavement Quality Concrete Design Lecture.pptx
byAditya Rajan Patra
 
PDF
Resonating 25 years of service- 1988 exam batch.pdf
byRajesh Prasad
 
PPT
Unit 2 Gas and Diesel Power Plant and working
byRoeverDirectoroffice
 
PDF
A Guide to Boiler Construction and Design
byMahmoud Moghtaderi
 
PDF
Computer Aided Engineering Graphics Notes till Mid-I Syllabus
byC Sai Kiran
 
PPTX
Importance of Management of Change | MaintWiz CMMS
byMaintWiz Technologies Private Limited
 
PDF
A Guide to Boiler Operation and Maintenance
byMahmoud Moghtaderi
 
PPTX
Refrigeration and air conditioning .pptx
bykhushicapricorn1001
 
PPTX
Strategic_Observability_How_to_Design_the_Right_Stack_for_Your_Organization.pptx
byMoustafa Mahmoud AboElnaga
 
PDF
13 Malpractices In Bunkering Operations Seafarers Should Be Aware Of
byMahmoud Moghtaderi
 
19ECX25-CAD for VLSI Circuits Unit-1.pptx
bytamil arasan
 
Factory_Automation used in Various Factories_Unit2_Detailed.pptx
bybekhem2008
 
Unit 1_Introduction to Automation & Robotics.pptx
bybekhem2008
 
2025 MediaEval Medico Organizer Presentation
bySushant Gautam
 
Operating System Concepts-Linux and Shell Programming by K. Adi
byAdiseshaK
 
First law of thermodynamics for chemical engineering.pptx
byAnonymousEBReO8v
 
19EC704 Optical and Microwave Lab Manual.docx
bytamil arasan
 
EcoSath Hackathon Final Presentation Based on ESG Data
byshashirwr2016
 
23AE205-IMAGE AND VLSI SIGNAL PROCESSING LAB.docx
bytamil arasan
 
From Orchestration to Platform The Missing Puzzles to Unify Polyglot Services...
byRichárd Kovács
 
Pavement Quality Concrete Design Lecture.pptx
byAditya Rajan Patra
 
Resonating 25 years of service- 1988 exam batch.pdf
byRajesh Prasad
 
Unit 2 Gas and Diesel Power Plant and working
byRoeverDirectoroffice
 
A Guide to Boiler Construction and Design
byMahmoud Moghtaderi
 
Computer Aided Engineering Graphics Notes till Mid-I Syllabus
byC Sai Kiran
 
Importance of Management of Change | MaintWiz CMMS
byMaintWiz Technologies Private Limited
 
A Guide to Boiler Operation and Maintenance
byMahmoud Moghtaderi
 
Refrigeration and air conditioning .pptx
bykhushicapricorn1001
 
Strategic_Observability_How_to_Design_the_Right_Stack_for_Your_Organization.pptx
byMoustafa Mahmoud AboElnaga
 
13 Malpractices In Bunkering Operations Seafarers Should Be Aware Of
byMahmoud Moghtaderi
 
In this document
Powered by AI

Introduction to the course on neural computation focusing on Multi-Layer Perceptron (MLP) and Backpropagation.

Definition of perceptrons, MLP architecture, including input, hidden, and output layers, and learning method.

MLPs handle non-linearly separable patterns and evolved due to backpropagation training algorithms.

Introduction and explanation of the backpropagation training mechanism, modifying synaptic weights.

Architecture overview with requirements like training set, testing set, and learning rate.

Three steps of backpropagation training—feedforward input, propagate error, adjust weights.

Explanation of key terms like input training vector, output target vector, and delta error.

Details on activation functions, expectations from them such as continuity and ease of computation.

Importance of generalization, methods to improve generalization, and factors like training set variety.

Demonstration of solving XOR problem using backpropagation, including architecture, weights, and training.

Weight updates during training, indicating modifications after iterations in learning.

Functional approximation capabilities of MLPs and requirements for hidden units based on functions.

Diverse applications of MLPs, including NETtalk, sonar classification, navigation, and OCR.

Techniques for enhancing BP performance, including learning rate adjustments and batch learning.

Overview of implementing a neuron-based architecture in MATLAB, detailing input and output processing.

Mathematical foundations behind BP learning rules, including delta rule derivation for hidden and output layers.

References for further study, including key textbooks and other educational materials.

Artificial Neural Networks Lect5: Multi-Layer Perceptron & Backpropagation

  • 1.
    1 CS407 Neural Computation Lecture5: The Multi-Layer Perceptron (MLP) and Backpropagation Lecturer: A/Prof. M. Bennamoun
  • 2.
    2 What is aperceptron and what is a Multi-Layer Perceptron (MLP)?
  • 3.
    3 What is aperceptron? wk1 x1 wk2 x2 wkm xm ... ... Σ Bias bk ϕ(.) vk Input signal Synaptic weights Summing junction Activation function Output yk bxwv kj m j kjk += ∑=1 )(vy kk ϕ= )()( ⋅=⋅ signϕ Discrete Perceptron: shapeS −=⋅)(ϕ Continous Perceptron:
  • 4.
    4 Activation Function ofa perceptron vi +1 -1 Signum Function (sign) )()( ⋅=⋅ signϕ Discrete Perceptron: shapesv −=)(ϕ Continous Perceptron: vi +1
  • 5.
    5 MLP Architecture The Multi-Layer-Perceptronwas first introduced by M. Minsky and S. Papert in 1969 Type: Feedforward Neuron layers: 1 input layer 1 or more hidden layers 1 output layer Learning Method: Supervised
  • 6.
    6 Terminology/Conventions Arrows indicate thedirection of data flow. The first layer, termed input layer, just contains the input vector and does not perform any computations. The second layer, termed hidden layer, receives input from the input layer and sends its output to the output layer. After applying their activation function, the neurons in the output layer contain the output vector.
  • 7.
    7 Why the MLP? Thesingle-layer perceptron classifiers discussed previously can only deal with linearly separable sets of patterns. The multilayer networks to be introduced here are the most widespread neural network architecture – Made useful until the 1980s, because of lack of efficient training algorithms (McClelland and Rumelhart 1986) – The introduction of the backpropagation training algorithm.
  • 8.
    8 Different Non-Linearly Separable Problemshttp://www.zsolutions.com/light.htm Structure Types of Decision Regions Exclusive-OR Problem Classes with Meshed regions Most General Region Shapes Single-Layer Two-Layer Three-Layer Half Plane Bounded By Hyperplane Convex Open Or Closed Regions Arbitrary (Complexity Limited by No. of Nodes) A AB B A AB B A AB B B A B A B A
  • 9.
    9 What is backpropagationTraining and how does it work?
  • 10.
    10 Supervised Error Back-propagationTraining – The mechanism of backward error transmission (delta learning rule) is used to modify the synaptic weights of the internal (hidden) and output layers • The mapping error can be propagated into hidden layers – Can implement arbitrary complex/output mappings or decision surfaces to separate pattern classes • For which, the explicit derivation of mappings and discovery of relationships is almost impossible – Produce surprising results and generalizations What is Backpropagation?
  • 11.
    11 Architecture: Backpropagation Network TheBackpropagation Net was first introduced by G.E. Hinton, E. Rumelhart and R.J. Williams in 1986 Type: Feedforward Neuron layers: 1 input layer 1 or more hidden layers 1 output layer Learning Method: Supervised Reference: Clara Boyd
  • 12.
    12 Backpropagation Preparation Training Set Acollection of input-output patterns that are used to train the network Testing Set A collection of input-output patterns that are used to assess network performance Learning Rate-α A scalar parameter, analogous to step size in numerical integration, used to set the rate of adjustments
  • 13.
    13 Backpropagation training cycle 1/Feedforward of the input training pattern 2/ Backpropagation of the associated error3/ Adjustement of the weights Reference Eric Plammer
  • 14.
    14 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 15.
    15 Source: Fausett, L.,Fundamentals of Neural Networks, Prentice Hall, 1994. Notation -- p. 292 of FausettNotation -- p. 292 of Fausett BP NN With Single Hidden Layer kjw , jiv , I/P layer O/P layer Hidden layer Reference: Dan St. Clair Fausett: Chapter 6
  • 16.
    16 Notation x = inputtraining vector t = Output target vector. δk = portion of error correction weight for wjk that is due to an error at output unit Yk; also the information about the error at unit Yk that is propagated back to the hidden units that feed into unit Yk δj = portion of error correction weight for vjk that is due to the backpropagation of error information from the output layer to the hidden unit Zj α = learning rate. voj = bias on hidden unit j wok = bias on output unit k
  • 17.
    17 Source: Fausett, L.,Fundamentals of Neural Networks, Prentice Hall, 1994. Hyberbolic tangent Binary step Activation Functions )](1[*)()( )exp(1 1 )( ' xfxfxf x xf −= −+ = Should be continuos, differentiable, and monotonically non-decreasing. Plus, its derivative should be easy to compute.
  • 18.
    18 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 19.
    19 Fausett, L., pp.294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
  • 20.
    20 Fausett, L., pp.294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
  • 21.
    21 Fausett, L., pp.294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
  • 22.
    22 Fausett, L., pp.294-296. Yk Z1 Zj Z3 X1 X2 X3 1 1
  • 23.
    23 Let’s examine Training AlgorithmEquations [ ]nxxX ...1= [ ]pvvV ,01,00 ...= Y1 Z1 Z2 Z3 X1 X2 X3 1 1 v2,1 Vectors & matrices make computation easier.           = pnn p vv vv V ,1, ,11,1 ... ......... ... [ ])_(...)_( _ 1 0 pinzfinzfZ XVVinZ = += Step 4 computation becomes [ ]mwwW ,01,00 ...=           = mpp m ww ww W ,1, ,11,1 ... ......... ... Step 5 computation becomes [ ])_(...)_( _ 1 0 minyfinyfY ZWWinY = +=
  • 24.
    24 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 25.
    25 Generalisation Once trained, weightsare held contstant, and input patterns are applied in feedforward mode. - Commonly called “recall mode”. We wish network to “generalize”, i.e. to make sensible choices about input vectors which are not in the training set Commonly we check generalization of a network by dividing known patterns into a training set, used to adjust weights, and a test set, used to evaluate performance of trained network
  • 26.
    26 Generalisation … Generalisation canbe improved by – Using a smaller number of hidden units (network must learn the rule, not just the examples) – Not overtraining (occasionally check that error on test set is not increasing) – Ensuring training set includes a good mixture of examples No good rule for deciding upon good network size (# of layers, # units per layer) Usually use one input/output per class rather than a continuous variable or binary encoding
  • 27.
    27 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 28.
    28 Example 1 The XORfunction could not be solved by a single layer perceptron network The function is: X Y F 0 0 0 0 1 1 1 0 1 1 1 0 Reference: R. Spillman
  • 29.
    29 XOR Architecture x y fv11 Σ v01 v21 fv12Σ v02 v22 fw11 Σ w01 w21 1 1 1
  • 30.
    30 Initial Weights Randomly assignsmall weight values: x y f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1
  • 31.
    31 Feedfoward – 1stPass x y f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1 Training Case: (0 0 0) 0 0 1 1 zin1 = -.3(1) + .21(0) + .15(0) = -.3 Activation function f: z1 = .43 zin2 = .25(1) -.4(0) + .1(0) z2 = .56 1 yin1 = -.4(1) - .2(.43) +.3(.56) = -.318 y1 = .42 (not 0) x e xf − + = 1 1 )(
  • 32.
    32 Backpropagate 0 0 f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2Σ -.4 .3 1 1 1 δ1 = (t1 – y1)f’(y_in1) =(t1 – y1)f(y_in1)[1- f(y_in1)] δ1 = (0 – .42).42[1-.42] = -.102 δ_in1 = δ1w11 = -.102(-.2) = .02 δ1 = δ_in1f’(z_in1) = .02(.43)(1-.43) = .005 δ_in2 = δ1w21 = -.102(.3) = -.03 δ2 = δ_in2f’(z_in2) = -.03(.56)(1-.56) = -.007
  • 33.
    33 Calculate the Weights– First Pass 0 0 f.21 Σ -.3 .15 f-.4 Σ .25 .1 f-.2 Σ -.4 .3 1 1 1 10 11 2,1 αδ αδ =∆ ==∆ w jzw jj jj ijij v jxv αδ αδ =∆ ==∆ 0 2,1 102.0 −=∆w 0571.)56)(.102.(2121 −=−==∆ zw δ 0439.)43)(.102.(1111 −=−==∆ zw δ 005.01 −=∆v 007.02 −=∆v 0)0)(005(.1111 ===∆ xv δ 0)0)(007.(1212 =−==∆ xv δ 0)0)(005(.2121 ===∆ xv δ 0)0)(007.(2222 =−==∆ xv δ
  • 34.
    34 Update the Weights– First Pass 0 0 f.21 Σ -.305 .15 f-.4 Σ .243 .1 f-.044Σ -.502 .243 1 1 1
  • 35.
    35 Final Result After about500 iterations: x y f1 Σ -1.5 1 f1 Σ -.5 1 f-2 Σ -.5 1 1 1 1
  • 36.
    36 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 37.
    37 Example 2 [ ]08.06.0=X          − = 2 1 1 w []10 −=w[ ]1000 −=v           = 130 221 012 v 9.0=t Y1 Z1 Z2 Z3 X1 X2 X3 1 1 v2,1 α = 0.3 Desired output for X input )1( 1 )( x e xf − + = m = 1 p = 3 n = 3 Reference: Vamsi Pegatraju and Aparna Patsa
  • 38.
    38 Primary Values: Inputsto Epoch - I X=[0.6 0.8 0]; W=[-1 1 2]’; W0=[-1]; V= 2 1 0 1 2 2 0 3 1 V0=[ 0 0 -1]; Target t=0.9; α = 0.3;
  • 39.
    39 Epoch – I Step4: Z_in= V0+XV = [ 2 2.2 0.6]; Z=f([Z_in])=[ 0.8808 0.9002 0.646]; Step 5: Y_in = W0+ZW = [0.3114]; Y=f([Z_in])=0.5772; Sum of Squares Error obtained originally: (0.9 – 0.5772)2 = 0.1042
  • 40.
    40 Step 6: Error= tk – Yk = 0.9 – 0.5772 Now we have only one output and hence the value of k=1. δ1= (t1 – y1 )f’(Y_in1) We know f’(x) for sigmoid = f(x)(1-f(x)) ⇒ δ1 = (0.9 −0.5772)(0.5772)(1−0.5772) = 0.0788
  • 41.
    41 For intermediate weightswe have (j=1,2,3) ∆Wj,k=α δκΖj = α δ1Ζj ⇒ ∆W1=(0.3)(0.0788)[0.8808 0.9002 0.646]’ =[0.0208 0.0213 0.0153]’; Bias ∆W0,1=α δ1= (0.3)(0.0788)=0.0236;
  • 42.
    42 Step 7: Backpropagationto the first hidden layer For Zj (j=1,2,3), we have δ_inj = ∑k=1..m δκWj,k= δ1Wj,1 ⇒ δ_in1=-0.0788;δ_in2=0.0788;δ_in3=0.1576; δj= δ_injf’(Z_inj) => δ1=-0.0083; δ2=0.0071; δ3=0.0361;
  • 43.
    43 ∆Vi,j = αδjXi ⇒∆V1 = [-0.0015 -0.0020 0]’; ⇒ ∆V2 = [0.0013 0.0017 0]’; ⇒ ∆V3 = [0.0065 0.0087 0]’; ∆V0=α[δ1 δ2 δ3] = [ -0.0025 0.0021 0.0108]; X=[0.6 0.8 0]
  • 44.
    44 Step 8: Updatingof W1, V1, W0, V0 Wnew= Wold+∆W1=[-0.9792 1.0213 2.0153]’; Vnew= Vold+∆V1 =[1.9985 1.0013 0.065;0.998 2.0017 2.0087; 0 3 1]; W0new = -0.9764; V0new = [-0.0025 0.0021 -0.9892]; Completion of the first epoch.
  • 45.
    45 Primary Values: Inputsto Epoch - 2 X=[0.6 0.8 0]; W=[-0.9792 1.0213 2.0153]’; W0=[-0.9792]; V=[1.9985 1.0013 0.065; 0.998 2.0017 2.0087; 0 3 1]; V0=[ -0.0025 0.0021 -0.9892]; Target t=0.9; α = 0.3;
  • 46.
    46 Epoch – 2 Step4: Z_in=V0+XV=[1.995 2.2042 0.6217]; Z=f([Z_in])=[ 0.8803 0.9006 0.6506]; Step 5: Y_in = W0+ZW = [0.3925]; Y=f([Z_in])=0.5969; Sum of Squares Error obtained from first epoch: (0.9 – 0.5969)2 = 0.0918
  • 47.
    47 Step 6: Error= tk – Yk = 0.9 – 0.5969 Now again, as we have only one output, the value of k=1. δ1= (t1 – y1 )f’(Y_in1) =>δ1 = (0.9 −0.5969)(0.5969)(1−0.5969) = 0.0729
  • 48.
    48 For intermediate weightswe have (j=1,2,3) ∆Wj,k=α δκΖj = α δ1Ζj ⇒ ∆W1=(0.3)*(0.0729)* [0.8803 0.9006 0.6506]’ =[0.0173 0.0197 0.0142]’; Bias ∆W0,1=α δ1= 0.0219;
  • 49.
    49 Step 7: Backpropagationto the first hidden layer For Zj (j=1,2,3), we have δ_inj = ∑k=1..m δκWj,k= δ1Wj,1 ⇒ δ_in1=-0.0714;δ_in2=0.0745;δ_in3=0.1469; δj= δ_injf’(Z_inj) => δ1=-0.0075; δ2=0.0067; δ3=0.0334;
  • 50.
    50 ∆Vi,j = αδjXi ⇒∆V1 = [-0.0013 -0.0018 0]’; ⇒ ∆V2 = [0.0012 0.0016 0]’; ⇒ ∆V3 = [0.006 0.008 0]’; ∆V0=α[δ1 δ2 δ3] = [ -0.0022 0.002 0.01];
  • 51.
    51 Step 8: Updatingof W1, V1, W0, V0 Wnew= Wold+∆W1=[-0.9599 1.041 2.0295]’; Vnew= Vold+∆V1 =[1.9972 1.0025 0.0125; 0.9962 2.0033 2.0167; 0 3 1]; W0new = -0.9545; V0new = [-0.0047 0.0041 -0.9792]; Completion of the second epoch.
  • 52.
    52 Z_in=V0+XV=[1.9906 2.2082 0.6417]; =>Z=f([Z_in])=[0.8798 0.9010 0.6551]; Step 5: Y_in = W0+ZW = [0.4684]; => Y=f([Z_in])=0.6150; Sum of Squares Error at the end of the second epoch: (0.9 – 0.615)2 = 0.0812. From the last two values of Sum of Squares Error, we see that the value is gradually decreasing as the weights are getting updated.
  • 53.
    53 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 54.
    54 Functional Approximation Multi-Layer Perceptronscan approximate any continuous function by a two-layer network with squashing activation functions. If activation functions can vary with the function, can show that a n-input, m-output function requires at most 2n+1 hidden units. See Fausett: 6.3.2 for more details.
  • 55.
    55 Function Approximators Example: afunction h(x) approximated by H(w,x)
  • 56.
    56 Applications We look ata number of applications for backpropagation MLP’s. In each case we’ll examine –Problem to be solved –Architecture Used –Results Reference: J.Hertz, A. Krogh, R.G. Palmer, “Introduction to the Theory of Neural Computation”, Addison Wesley, 1991
  • 57.
    57 NETtalk - Specifications Problemis to convert written text to speech. Conventionally, this is done by hand-coded linguistic rules, such as the DECtalk system. NETtalk uses a neural network to achieve similar results Input is written text Output is choice of phoneme for speech synthesiser
  • 58.
    58 NETtalk - architecture ec ah t oT n 7 letter sliding window, generating phoneme for centre character. Input units use 1 of 29 code. => 203 input units (=29x7) 80 hidden units, fully interconnected 26 output units, 1 of 26 code representing most likely phoneme
  • 59.
    59 NETtalk - Results 1024Training Set After 10 epochs - intelligible speech After 50 epochs - 95% correct on training set - 78% correct on test set Note that this network must generalise - many input combinations are not in training set Results not as good as DECtalk, but significantly less effort to code up.
  • 60.
    60 Sonar Classifier Task -distinguish between rock and metal cylinder from sonar return of bottom of bay Convert time-varying input signal to frequency domain to reduce input dimension. (This is a linear transform and could be done with a fixed weight neural network.) Used a 60-x-2 network with x from 0 to 24 Training took about 200 epochs. 60-2 classified about 80% of training set; 60-12-2 classified 100% training, 85% test set
  • 61.
    61 ALVINN Drives 70 mphon a public highway 30x32 pixels as inputs 30 outputs for steering 30x32 weights into one out of four hidden unit 4 hidden units
  • 62.
    62 Navigation of aCar Task is to control a car on a winding road Inputs are a 30x32 pixel image from a video camera on roof, 8x32 image from a range finder => 1216 inputs 29 hidden units 45 output units arranged in a line, 1-of-45 code representing hard-left..straight-ahead..hard-right
  • 63.
    63 Navigation of Car- Results Training set of 1200 simulated road images Trained for 40 epochs Could drive at 5 km/hr on road, limited by calculation speed of feed-forward network. Twice as fast as best non-net solution
  • 64.
    64 Backgammon Trained on 3000example board scenarios of (position, dice, move) rated from -100 (very bad) to +100 (very good) from human expert. Some important information such as “pip- count” and “degree-of-trapping” was included as input. Some “noise” added to input set (scenarios with random score) Handcrafted examples added to training set to correct obvious errors
  • 65.
    65 Backgammon results 459 inputs,2 hidden layers, each 24 units, plus 1 output for score (All possible moves evaluated) Won 59% against a conventional backgammon program (41% without extra info, 45% without noise in training set) Won computer olympiad, 1989, but lost to human expert (Not surprising since trained by human scored examples)
  • 66.
    66 Encoder / ImageCompression Wish to encode a number of input patterns in an efficient number of bits for storage or transmission We can use an autoassociative network, i.e. an M-N-M network, where we have M inputs, and N<M hidden units, M outputs, trained with target outputs same as inputs Hidden units need to encode inputs in fewer signals in the hidden layers. Outputs from hidden layer are encoded signal
  • 67.
    67 Encoders We can store/transmithidden values using first half of network; decode using second half. We may need to truncate hidden unit values to fixed precision, which must be considered during training. Cottrell et al. tried 8x8 blocks (8 bits each) of images, encoded in 16 units, giving results similar to conventional approaches. Works best with similar images
  • 68.
    68 Neural network forOCR feedforward network trained using Back- propagation A B E D C Output Layer Input Layer Hidden Layer 8 10 8 8 1010
  • 69.
    69 Pattern Recognition Post-code (orZIP code) recognition is a good example - hand-written characters need to be classified. One interesting network used 16x16 pixel map input of handwritten digits already found and scaled by another system. 3 hidden layers plus 1-of-10 output layer. First two hidden layers were feature detectors.
  • 70.
    70 ZIP code classifier Firstlayer had same feature detector connected to 5x5 blocks of input, at 2 pixel intervals => 8x8 array of same detector, each with the same weights but connected to different parts of input. Twelve such feature detector arrays. Same for second hidden layer, but 4x4 arrays connected to 5x5 blocks of first hidden layer; with 12 different features. Conventional 30 unit 3rd hidden layer
  • 71.
    71 ZIP Code Classifier- Results Note 8x8 and 4x4 arrays of feature detectors use the same weights => many fewer weights to train. Trained on 7300 digits, tested on 2000 Error rates: 1% on training, 5% on test set If cases with no clear winner rejected (i.e. largest output not much greater than second largest output), then, with 12% rejection, error rate on test set reduced to 1%. Performance improved further by removing more weights: “optimal brain damage”.
  • 72.
    72 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 73.
    73 Heuristics for makingBP Better Training with BP is more an art than science – result of own experience Normalizing the inputs – preprocessed so that its mean value is closer to zero (see “prestd” function in matlab). – input variables should be uncorrelated • by “Principal Component Analysis” (PCA). See “prepca” and “trapca” functions in Matlab.
  • 74.
    74 Sequential vs. Batchupdate “Sequential” learning means that a given input pattern is forward propagated, the error is determined and back-propagated, and the weights are updated. Then the same procedure is repeated for the next pattern. “Batch” learning means that the weights are updated only after the entire set of training patterns has been presented to the network. In other words, all patterns are forward propagated, and the error is determined and back-propagated, but the weights are only updated when all patterns have been processed. Thus, the weight update is only performed every epoch. If P = # patterns in one epoch ∑= ∆=∆ P p pw P w 1 1
  • 75.
    75 Sequential vs. Batchupdate i.e.in some cases, it is advantageous to accumulate the weight correction terms for several patterns (or even an entire epoch if there are not too many patterns) and make a single weight adjustment (equal to the average of the weight correction terms) for each weight rather than updating the weights after each pattern is presented. This procedure has a “smoothing effect” (because of the use of the average) on the correction terms. In some cases, this smoothing may increase the chances of convergence to a local minimum.
  • 76.
    76 Initial weights Initial weights– will influence whether the net reaches a global (or only a local minimum) of the error and if so, how quickly it converges. – The values for the initial weights must not be too large otherwise, the initial input signals to each hidden or output unit will be likely to fall in the region where the derivative of the sigmoid function has a very small value (f’(net)~0) : so called saturation region. – On the other hand, if the initial weights are too small, the net input to a hidden or output unit will be close to zero, which also causes extremely slow learning. – Best to set the initial weights (and biases) to random numbers between –0.5 and 0.5 (or between –1 and 1 or some other suitable interval). – The values may be +ve or –ve because the final weights after training may be of either sign also.
  • 77.
    77 Memorization vs. generalization Howlong to train the net: Since the usual motivation for applying a backprop net is to achieve a balance between memorization and generalization, it is not necessarily advantageous to continue training until the error actually reaches a minimum. – Use 2 disjoint sets of data during training: 1/ a set of training patterns and 2/ a set of training- testing patterns (or validation set). – Weight adjustment are based on the training patterns; however, at intervals during training, the error is computed using the validation patterns. – As long as the error for the validation decreases, training continues. – When the error begins to increase, the net is starting to memorize the training patterns too specifically (starts to loose its ability to generalize). At this point, training is terminated.
  • 78.
    78 Early stopping Error Training time Withtraining set (which changes wij) With validation set (which does not change wij) Stop Here ! L. Studer, IPHE-UNIL
  • 79.
    79 Backpropagation with momentum Backpropagationwith momentum: the weight change is in a direction that is a combination of 1/ the current gradient and 2/ the previous gradient. Momentum can be added so weights tend to change more quickly if changing in the same direction for several training cycles:- ∆ wij (t+1) = α δ xi + µ . ∆ wij (t) µ is called the “momentum factor” and ranges from 0 < µ < 1. – When subsequent changes are in the same direction increase the rate (accelerated descent) – When subsequent changes are in opposite directions decrease the rate (stabilizes)
  • 80.
    80 Backpropagation with momentum… Weightupdate equation Momentum )1( −tw )(tw z)( αδ+tw )1( +tw Source: Fausett, L., Fundamentals of Neural Networks, Prentice Hall, 1994, pg. 305.
  • 81.
    81 Source: Fausett, L.,Fundamentals of Neural Networks, Prentice Hall, 1994. Adaptive learning rate BP training algorithm – Adaptive Learning Rate BP training algorithm – Adaptive Learning Rate
  • 82.
    82 Adaptive Learning rate… AdaptiveParameters: Vary the learning rate during training, accelerating learning slowly if all is well ( error, E, decreasing) , but reducing it quickly if things go unstable (E increasing). For example: Typically, a = 0.1, b = 0.5      >∆ <∆+ =+ otherwise(t) 0Eif(t).b)-(1 epochsfewlastfor0Eifa(t) 1)(t α α α α
  • 83.
    83 Matlab BP NNArchitecture A neuron with a single R-element input vector is shown below. Here the individual element inputs • are multiplied by weights • and the weighted values are fed to the summing junction. Their sum is simply Wp, the dot product of the (single row) matrix W and the vector p. The neuron has a bias b, which is summed with the weighted inputs to form the net input n. This sum, n, is the argument of the transfer function f. • This expression can, of course, be written in MATLAB code as: •n = W*p + b However, the user will seldom be writing code at this low level, for such code is already built into functions to define and simulate entire networks.
  • 84.
    84 Matlab BP NNArchitecture
  • 85.
    85 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 86.
    86 Learning Rule Similar toDelta Rule. Our goal is to minimize the error, E, which is the difference between targets, tm , and our outputs, yk m , using a least squares error measure: E = 1/2 Σk (tk - yk)2 To find out how to change wjk and vij to reduce E, we need to find ijjk v E and w E ∂ ∂ ∂ ∂ Fausett, section 6.3, p324
  • 87.
    87 Delta Rule DerivationHidden-to-Output       −=−= ∑∑ k 2 kk jkjk 2 )y(t 2 1 ww E hence][5.0 ∂ ∂ ∂ ∂ k kk ytE [ ] [ ]       −=      −= ∑ 2 K JKk 2 k JKJK )(t 2 1 w t 2 1 ww E inKk yfy ∂ ∂ ∂ ∂ ∂ ∂ and)(where ∑== j jKjinKinKk wzyyfy JKinKK JK ).z(y')fy(t w E −−= ∂ ∂ JK inK JK inK w y w yf ∂ ∂ −−= ∂ ∂ −−= )( ).(y')fy(t )( )y(t w E KinKKKK JK∂ ∂ Notice the difference between the subscripts k (which corresponds to any node between hidden and output layers) and K (which represents a particular node K of interest)
  • 88.
    88 Delta Rule DerivationHidden-to-Output )(y')f(t:definetoconvenientisIt inKK KK y−=δ jkjinkkk zzyfyt δαα ∂ ∂ α =−=−=∆ )('][ w E wThus, jk jk jk zδα=∆ jkwsummary,In )(y')f(twith inKK KK y−=δ
  • 89.
    89 Delta Rule Derivation:Input to Hidden IJ ink ink k kk IJ k k kk v y yfyt v y yt v ∂ ∂ −−= ∂ ∂ −−= ∑∑ )('][][ E IJ∂ ∂ ])[(' v E IJ IinJJk k k IJ J Jk k k IJ ink k k xzfw v z w v y ∑∑∑ −= ∂ ∂ −= ∂ ∂ −= δδδ ∂ ∂       −=−= ∑∑ k 2 kk IJIJ 2 )y(t 2 1 v E hence][5.0 v ytE k kk ∂ ∂ ∂ ∂ and)(where ∑== j jKjinKinKk wzyyfy )(z'f:definetoconvenientisIt inJ∑= k JkkJ wδδ Notice the difference between the subscripts j and J and i and I ij k jkkiinjij xwxzfv αδδα ∂ ∂ α ==−=∆ ∑)(' v E ij
  • 90.
    90 Delta Rule Derivation:Input to Hidden )(z'f:where inJ∑= k JkkJ wδδ ijij xv αδ=∆summaryIn
  • 91.
    91 Backpropagation Neural Networks Architecture BPtraining Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading Architecture BP training Algorithm Generalization Examples – Example 1 – Example 2 Uses (applications) of BP networks Options/Variations on BP – Momentum – Sequential vs. batch – Adaptive learning rates Appendix References and suggested reading
  • 92.
    92 Suggested Reading. L. Fausett,“Fundamentals of Neural Networks”, Prentice-Hall, 1994, Chapter 6.
  • 93.
    93 References: These lecture noteswere based on the references of the previous slide, and the following references 1. Eric Plummer, University of Wyoming www.karlbranting.net/papers/plummer/Pres.ppt 2. Clara Boyd, Columbia Univ. N.Y comet.ctr.columbia.edu/courses/elen_e4011/2002/Artificial.ppt 3. Dan St. Clair, University of Missori-Rolla, http://web.umr.edu/~stclair/class/classfiles/cs404_fs02/Misc/CS 404_fall2001/Lectures/Lect09_102301/ 4. Vamsi Pegatraju and Aparna Patsa: web.umr.edu/~stclair/class/classfiles/cs404_fs02/ Lectures/Lect09_102902/Lect8_Homework/L8_3.ppt 5. Richard Spillman, Pacific Lutheran University: www.cs.plu.edu/courses/csce436/notes/pr_l22_nn5.ppt 6. Khurshid Ahmad and Matthew Casey Univ. Surrey, http://www.computing.surrey.ac.uk/courses/cs365/