7COM1033: Neural Networks and Machine Learning

7th May 2021, 10:00 - 11:30

Time: 90 minutes
Total marks: 82, 5 questions, 4 pages of questions
Version 0000

1. Associative Networks (TOTAL: 21 marks)

(a) (4 points) Give an example of a hetero-associative memory task you might encounter in real life.

(b) (4 points) The following diagram shows an associative (Willshaw) net with four input units and
four output units. In this network, 15 of the 16 synapses are switched off. The only synapse that
is switched on is marked by a black dot. The association between which input pattern and which
output pattern has been stored?

(c) (7 points) Store the following input-output pattern pairs in the net: input pattern 1 = (1, 1, 0, 0),
output pattern 1 = (0, 0, 1, 1); input pattern 2= (1, 0, 1, 0), output pattern 2 = (0, 1, 0, 1). Redraw
the net and label the synapses that have been switched on.

(d) (6 points) Present input pattern 1 and try to recall the correct output pattern. What are the input
sum, threshold and resulting activation of each of the four output units?
2. Perceptrons (TOTAL: 15 marks)
(a) (4 points) In a simple threshold linear unit (perceptron) with two inputs x1 and x2 and weights w1
and w2 , how is the output y calculated?

(b) (5 points) b) Draw the input space (including all four input patterns with an indication of their
target outputs and a potential decision line) for the following classification task:

x1 x2 target output
0 1 0
0.5 0 0
0 2 1
1.5 0 1

(c) (3 points) Give the equation that describes your decision surface .

(d) (3 points) Give potential weights w1 and w2 and a potential threshold θ for the decision surface

3. Unsupervised Learning (TOTAL: 10 marks)

Assume a Kohonen Network (Self-Organising Map) with two input units and a physical grid of 3 × 3 = 9
output units a, b, c, d, e, f , g, h, i. The weight vectors of the output units are initially given by (shown
by their physical arrangement in output space):

wa = (1, 5) wa = (2, 3) wc = (4, 1)

wd = (5, 1) we = (3, 2) wf = (1, 4)
wg = (5, 5) wh = (3, 3) wi = (1, 1)

(a) (4 points) The network is presented with an input vector x = (7, 1). Which output unit will be the
winner and why?

(b) (2 points) Assuming a learning rate α = 0.5, what will the weight vector of the winning output be
changed to after a single presentation of the input vector x = (7, 1)?

(c) (4 points) The network from (a) is presented with a large number of inputs where x1 and x2 are
chosen randomly from values between 0 and 10. Draw the final state of the network both in
(mathematical) input space and in (physical) output space.

Page 2
4. Deep Learning (TOTAL: 15 points)
(a) Consider an RGB input image with shape 100 × 200 × 3.
i. (3 points) If you flatten this image and connect it to 20 hidden units in the next layer of
a feedforward neural network, how many parameters are needed to describe the relationship
between these two layers (assuming no bias vector)? Explain how you arrive at your answer.

ii. (3 points) If instead you run this image in a convolutional layer with filters of size 10 × 10,
assuming a stride length of 1, how many times can you slide the filter in the vertical and
horizontal directions? Finally, given your previous answers, what is the size of the resulting
filter maps?

(b) Let A, the 5×5 matrix below, represent the pixel values of a simplified input image in a convolutional
neural network.
 
20 0 0 0 10
 7 20 0 0 10
 
20 0
A= 20 0 10

20 0 0 20 10
20 0 0 0 20
Assuming no padding, a stride length of 1 and a bias of 0, calculate the feature maps resulting from
the following filters:
 
1 0 0
i. (2 points) F1 = 1 0 0
1 0 0

 
1 0 0
ii. (2 points) F2 = 1 1 0
1 0 1

 
1 0 1
iii. (2 points) F3 = 0 1 1
0 0 1

iv. (3 points) Interpret the above results (i.e. describe the kinds of features in an image each filter
emphasizes, e.g. corners).

Page 3
5. Deep Learning (TOTAL: 21 points)
(a) (2 points) Find the derivative σ 0 (z) of the sigmoid function given below:
1
σ(z) =
1 + e−z
Given your answer in terms of e and z.

(b) (1 point) Which of the following is true of σ(z) when z becomes very large and positive?
i. σ(z) approaches ∞.
ii. σ(z) approaches 0.
iii. σ(z) approaches 1.

(c) (3 points) Given the relationship σ 0 (z) = σ(z)(1 − σ(z)), what happens to σ 0 (z) as ||z|| gets large?
Explain your answer.

(d) A very simple neural network is given by the following set of equations:

z1 = w1 x
a1 = ReLU (z1 )
z2 = w2 a1 + b
y = σ(z2 )

The loss function is given by C = 21 (y − t)2 , where y is the prediction and t is the target. The ReLU
function is defined as ReLU (z) = max(0, z) for z ∈ R.
i. (4 points) How many neurons and layers are there in this network? What are the activation
functions?

ii. (2 points) What is the output of the network, y, if the input x = 1 , and the weights and biases
are w1 = 1,w2 = 1,b = 2?

iii. (1 point) What is the loss for the input values given in part ii) and t = 1? (You may give your
answer to 2 significant figures).

∂C
iv. (5 points) Use the Chain rule to give an expression for ∂w1 .

v. (3 points) Name and describe a problem often encountered in neural networks that occurs when
many partial derivatives are multiplied together. How is training affected?

Page 4