Scribd
Upload
409 views

A Gentle Introduction To Neural Networks With Python

This document provides an introduction to neural networks using Python. It begins with basic concepts like predicting kilometers to miles conversions. It then covers more complex topics like classifying bugs based on their width and length. Key points are made throughout, such as using multiple nodes to solve problems that single nodes cannot. Finally, it demonstrates a live neural network that can recognize handwritten digits with over 98% accuracy.

Uploaded by

bobby
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
409 views

A Gentle Introduction To Neural Networks With Python

This document provides an introduction to neural networks using Python. It begins with basic concepts like predicting kilometers to miles conversions. It then covers more complex topics like classifying bugs based on their width and length. Key points are made throughout, such as using multiple nodes to solve problems that single nodes cannot. Finally, it demonstrates a live neural network that can recognize handwritten digits with over 98% accuracy.

Uploaded by

bobby
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 85

A Gentle Introduction

to Neural Networks
(with Python)
Tariq Rashid @postenterprise
EuroPython Bilbao July 2016

Background
Ideas
DIY
Handwriting
Thoughts

and a

live demo!

Background

Start With Two Questions

locate people in this photo

add these numbers

+
+
+
+
+
+

2403343781289312
2843033712837981
2362142787897881
3256541312323213
9864479802118978
8976677987987897
8981257890087988
= ?

AI is Huge!

Googles and Go

Ideas

Simple Predicting Machine

Simple Predicting Machine

Kilometres to Miles

random starting parameter


try a model - this one is linear

Kilometres to Miles

not great

Kilometres to Miles

better

Kilometres to Miles

worse !

Kilometres to Miles

best yet !

Key Points

1.

2.

xactly? Try
e
s
rk
o
w
g
in
how someth
w
o
n
k
.
t
n
o
D
parameters
le
b
ta
s
ju
d
a
model with
.
parameters
e
th
e
n
fi
re
or to
Use the err

Garden Bugs

Classifying Bugs

Classifying Bugs

Classifying Bugs

Classifying Bugs

Key Points

ings.

1.

Classifying

thing

th
predicting
e
k
li
a
d
in
k
s is

Learning from Data

Example

Width

Length

Bug

3.0

1.0

ladybird

1.0

3.0

caterpillar

Learning from Data

Learning from Data

not a good separator

Learning from Data

shift the line up


just above the training data point

Learning from Data

How Do We Update The Parameter?

error = target - actual


E = (A + A)x - Ax
A = E / x

Hang On!

Oh no!
each update ignores
previous examples

Calm Down the Learning

A = L (E / x)

learning rate

Calm Down the Learning

learning rate = 0.5

Key Points

1.

you
d - ensures
o
o
g
is
g
in
your learn
pact of
im
s
e
c
u
d
Moderating
re
nd
your data, a
ll
a
m
o
fr
rn
lea
.
training data
y
is
o
n
r
o
outliers

Boolean Logic

IF I have eaten my vegetables AND I am still hungry


THEN I can have ice cream.

IF its the weekend OR I am on annual leave THEN Ill


go to the park.

Input A

Input B

AND

OR

Boolean Logic

Boolean Logic

XOR Puzzle!

Input A

Input B

XOR

XOR Solution!

Use more than one node!

Key Points

gle
ith just a sin
w
d
e
lv
o
s
e
nt b
roblems ca
p
e
m
o
S
1.
r classifier.
simple linea
g together to
in
rk
o
w
s
e
nod
e multiple
s
u
n
a
c
u
o
s.
2. Y
se problem
e
th
f
o
y
n
a
solve m

Brains in Nature

Brains in Nature

brain 0.4 grams

11,000 neurons

natures brains can eat, fly, navigate, fight,


communicate, play, learn
.. and theyre

resilient

302 neurons

37 billion neurons
(humans 20 billion)

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons
https://faculty.washington.edu/chudler/facts.html

Brains in Nature

Brains in Nature

logistic function
y = 1 / (1 + e-x)

Brains in Nature

Artificial Neuron

Artificial Neural Network .. finally!

Pause.

...

Where Does The Learning Happen?

link weight?

sigmoid function slope?

Key Points

things, and
d
te
a
c
ti
is
h
do sop
ct
brains can
l
ra
and imperfe
tu
e
a
g
a
N
m
a
d
1.
to
ly resilient
are incredib
omputing.
c
l
a
n
io
it
d
a
like tr
signals .. un
d
artly inspire
p
s
in
ra
b
l
a
logic
to copy bio
g
in
ry
T
.
2
etworks.
n
l
a
r
u
e
n
l
artificia
its
arameter p
le
b
ta
s
ju
e ad
ights are th
e
w
k
in
L
s.
3.
ing happen
rn
a
le
e
th
where

Feeding Signals Forward

Feeding Signals Forward

Feeding Signals Forward

Matrix Multiplication

Matrix Multiplication

weights

incoming signals

WI = X
dot product

Key Points

e
tions can b
la
u
lc
a
c
rd
a
no
iplication,
any feedforw
lt
m
u
e
m
h
T
ix
tr
a
.
1
as m
concisely
d
e
s
s
re
p
x
e
network.
e
th
e
p
a
h
s
t
matter wha
n do matrix
a
c
s
e
g
a
u
g
lan
.
rogramming
p
e
m
nd quickly
o
a
S
y
tl
n
2.
ie
ic
n really eff
multiplicatio

Network Error

Network Error

Internal Error

Internal Error

Matrices Again!

Key Points

we
guide how
to
r
o
r
r
e
e
th
ber we use
m
e
m
e
k weights.
R
n
li
.
r
1
te
e
m
dels para
refine a mo
e
is easy - th
s
e
d
o
n
t
u
tp
or at the ou
nd actual
a
d
e
ir
s
2. The err
e
d
etween the
difference b
outputs.
obvious. A
t
n
is
s
e
d
o
n
l
or at interna
portion to
rr
o
r
e
p
e
h
in
T
it
t
li
.
p
3
to s
pproach is
a
c
ti
is
r
u
e
h
hts.
the link weig
be
e error can
th
g
n
ti
a
g
a
back prop
too!
4. and
ultiplication
m
ix
tr
a
m
as a
expressed

Yes, But How Do We Actually Update The Weights?

Aaarrrggghhh !!

Perfect is the Enemy of Good

landscape is a complicated difficult mathematical function ..


with all kinds of lumps, bumps, kinks

Gradient Descent

smaller gradient .. youre closer to the


bottom take smaller steps?

Key Points

e
of finding th
y
a
w
l
a
c
ti
c
a pra
t descent is
n
ie
d
a
r
G
1.
nctions.
fu
lt
u
c
fi
if
d
f
minimum o
by
rshooting
e
v
o
f
o
e
c
n
ha
shallower.
avoid the c
ts
n
e
a
g
c
t
u
n
o
ie
Y
d
ra
2.
eg
r steps if th
e
ll
a
m
s
g
in
tak
tion
ifficult func
d
a
is
rk
o
etw
nt
f a neural n
o
r
o
rr
dient desce
e
ra
e
g
h
e
b
y
a
3. T
m
eights so
of the link w
will help ...

Climbing Down the Network Error Landscape

We need to find this gradient

Error Gradient

E = (desired - actual)2

school level calculus (chain rule)

dE/dwij = - ej . oj . (1 - oj) . oi
previous node

A gentle intro to calculus


http://makeyourownneuralnetwork.blogspot.co.uk/2016/01/a-gentle-introduction-to-calculus.html

Updating the Weights

move wjk in the opposite direction to


the slope

remember that learning rate

DIY

Python Class and Functions

Neural Network
Class

Initialise

Train

Query

set size, initial weights

do the learning

query for answers

Python has Cool Tools

matrix maths

numpy
scipy
matplotlib
notebook

Function - Initialise

# initialise the neural network


def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# set number of nodes in each input, hidden, output layer
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes
# link weight matrices, wih and who
# weights inside the arrays are w_i_j, where link is from node i to node j in the next layer
# w11 w21
# w12 w22 etc
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
# learning rate
self.lr = learningrate
# activation function is the sigmoid function
self.activation_function = lambda x: scipy.special.expit(x)
pass

numpy.random.normal()

random initial weights

Function - Query

combined weighted signals into hidden


layer
then sigmoid applied
# query the neural network
def query(self, inputs_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
# calculate signals into hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)
return final_outputs

numpy.dot()

similar for output layer

Function - Train

# train the neural network


def train(self, inputs_list, targets_list):
# convert inputs list to 2d array
inputs = numpy.array(inputs_list, ndmin=2).T
targets = numpy.array(targets_list, ndmin=2).T

same feed forward as before

# calculate signals into hidden layer


hidden_inputs = numpy.dot(self.wih, inputs)
# calculate the signals emerging from hidden layer
hidden_outputs = self.activation_function(hidden_inputs)
# calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)
# calculate the signals emerging from final output layer
final_outputs = self.activation_function(final_inputs)

output layer errors


hidden layer errors

# output layer error is the (target - actual)


output_errors = targets - final_outputs
# hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)
# update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.
transpose(hidden_outputs))
# update the weights for the links between the input and hidden layers
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.
transpose(inputs))
pass

update weights

Handwriting

Handwritten Numbers Challenge

MNIST Datasets

MNIST dataset:
60,000 training data examples
10,000 test data examples

MNIST Datasets

label
784 pixels
values

28 by 28 pixel image

Output Layer Values

Experiments

96% is very good!


weve only used simple ideas
and code
random processes
do go wonky!

More Experiments

98%

is amazing!

Thoughts

Peek Inside The Mind Of a Neural Network?

Peek Inside The Mind Of a Neural Network?

this isnt done very


often

Thanks!
live demo!

Finding Out More

makeyourownneuralnetwork.blogspot.co.uk

github.com/makeyourownneuralnetwork
www.amazon.co.uk/dp/B01EER4Z4G

twitter.com/myoneuralnet
slides goo.gl/JKsb62

Raspberry Pi Zero

It all works on a Raspberry Pi Zero


and it only costs 4 / $5 !!

You might also like