Uploaded byProfessor Lili Saghafi
Machine learning by using python lesson 2 Neural Networks By Professor Lili Saghafi
This document provides an overview of lesson 2 of a machine learning course using Python. It discusses neural networks and their biological inspiration. It then explains how artificial neural networks work, including the basic neuron structure and how signals are received and transmitted. Finally, it introduces implementing simple neural networks in Python using NumPy for efficient data structures.
More Related Content
Similar to Machine learning by using python lesson 2 Neural Networks By Professor Lili Saghafi
Machine learning by using python lesson 2 Neural Networks By Professor Lili Saghafi
- 1. 1 Machine Learning by usingPython Neural Network Lesson 2 By: Professor Lili Saghafi [email protected] @Lili_PLS
- 2. 2 Overview • Machine learningis the kind of programming which gives computers the capability to automatically learn from data without being explicitly programmed. • This means in other words that these programs change their behavior by learning from data. • In this course we will cover various aspects of machine learning • Everything will be related to Python. So it is Machine Learning by using Python. • What is the best programming language for machine learning? • Python is clearly one of the top players!
- 3. 3 Neural Networks • Whenwe say "Neural Networks", we mean artificial Neural Networks (ANN). The idea of ANN is based on biological neural networks like the brain. • The basic structure of a neural network is the neuron. A neuron in biology consists of three major parts: the soma (cell body), the dendrites, and the axon. • The dendrites branch of from the soma in a tree-like way and getting thinner with every branch. They receive signals (impulses) from other neurons at synapses. The axon - there is always only one - also leaves the soma and usually tend to extend for longer distances than the dentrites. The axon is used for sending the output of the neuron to other neurons or better to the synapsis of other neurons.
- 4.
- 5. 5 The following imageby Quasar Jarosz, courtesy of Wikipedia, illustrates this:
- 6. 6 already an abstractionfor a biologist, we can further abstract it:
- 7. 7 A perceptron ofartificial neural networks is simulating a biological neuron.
- 8. 8 Neural Networks • Itis amazingly simple, what is going on inside the body of a perceptron or neuron. The input signals get multiplied by weight values, i.e. each input has its corresponding weight. This way the input can be adjusted individually for every xixi. We can see all the inputs as an input vector and the corresponding weights as the weights vector.
- 9. 9 Neural Networks • Whena signal comes in, it gets multiplied by a weight value that is assigned to this particular input. That is, if a neuron has three inputs, then it has three weights that can be adjusted individually. The weights usually get adjusted during the learn phase. After this the modified input signals are summed up. It is also possible to add additionally a so-called bias b to this sum. The bias is a value which can also be adjusted during the learn phase. • Finally, the actual output has to be determined. For this purpose an activation or step function Φ is applied to weighted sum of the input values.
- 10. 10 the actual outputhas to be determined. For this purpose an activation or step function Φ is applied to weighted sum of the input values.
- 11. 11 The simplest formof an activation function is a binary function • The simplest form of an activation function is a binary function. If the result of the summation is greater than some threshold s, the result of Φ will be 1, otherwise 0.
- 12. 12 A Simple NeuralNetwork • The following image shows the general building principle of a simple artificial neural network: •
- 13. 13 • We willwrite a very simple Neural Network implementing the logical "And" and "Or" functions. • Let's start with the "And" function. It is defined for two inputs:
- 14. 14 Line Separation • Youcould imagine that you have two attributes describing am eddible object like a fruit for example: "sweetness" and "sourness" • We could describe this by points in a two- dimensional space. The x axis for the sweetness and the y axis for the sourness. Imagine now that we have two fruits as points in this space, i.e. an orange at position (3.5, 1.8) and a lemon at (1.1, 3.9).
- 15. 15 Line Separation • Wecould define dividing lines to define the points which are more lemon-like and which are more orange-like. The following program calculates and renders a bunch of lines. The red ones are completely unusable for this purpose, because they are not separating the classes. Yet, it is obvious that even the green ones are not all useful.
- 16.
- 17.
- 18.
- 19. 19 Neural Network UsingPython and Numpy
- 20. 20 Neural Network UsingPython and Numpy
- 21. 21 Neural Network UsingPython and Numpy • We have introduced the basic ideas about neuronal networks in the previous chapter of our tutorial. • We pointed out the similarity between neurons and neural networks in biology. We also introduced very small articial neural networks and introduced decision boundaries and the XOR problem. • The focus in our previous chapter had not been on efficiency.
- 22. 22 Neural Network UsingPython and Numpy • We will introduce a Neural Network class in Python in this chapter, which will use the powerful and efficient data structures of Numpy. This way, we get a more efficient network than in our previous chapter. When we say "more efficient", we do not mean that the artificial neural networks encountered in this lesson are efficient and ready for real life usage. • They are still quite slow compared to implementations from sklearn for example. The focus is to implement a very basic neural network and by doing this explaining the basic ideas.
- 23. 23 Neural Network UsingPython and Numpy • Ideas like how the signal flow inside of a network works, how to implement weights. how to initialize weight matrices or what activation functions can be used. • We will start with a simple neural networks consisting of three layers, i.e. the input layer, a hidden layer and an output layer.
- 24. 24 A Simple ArtificialNeural Network Structure • You can see a simple neural network structure in the following diagram. We have an input layer with three nodes i1,i2,i3i1,i2,i3 These nodes get the corresponding input values x1,x2,x3x1,x2,x3. The middle or hidden layer has four nodes h1,h2,h3,h4h1,h2,h3,h4. • The input of this layer stems from the input layer. We will discuss the mechanism soon. Finally, our output layer consists of the two nodes o1,o2
- 25. 25 Neural Network UsingPython and Numpy • We have to note that some would call this a two layer network, because they don't count the inputs as a layer.
- 26. 26 Neural Network UsingPython and Numpy • The input layer consists of the nodes i1i1, i2i2 and i3i3. In principle the input is a one-dimensional vector, like (2, 4, 11). A one-dimensional vector is represented in numpy like this:
- 27. 27 A Simple ArtificialNeural Network Structure
- 28. 28 A Simple ArtificialNeural Network Structure • In the algorithm, which we will write later, we will have to transpose it into a column vector, i.e. a two-dimensional array with just one column:
- 29. 29 A Simple ArtificialNeural Network Structure
- 30.
- 31.
- 32. 32 Backpropagation in Neural Networks •We have already written Neural Networks in Python in the previous chapters of our tutorial. We could train these networks, but we didn't explain the mechanism used for training. We used backpropagation without saying so. Backpropagation is a commonly used method for training artificial neural networks, especially deep neural networks.
- 33. 33 Backpropagation in Neural Networks •Backpropagation is needed to calculate the gradient, which we need to adapt the weights of the weight matrices. The weight of the neuron (nodes) of our network are adjusted by calculating the gradient of the loss function. For this purpose a gradient descent optimization algorithm is used. It is also called backward propagation of errors.
- 34. 34 Backpropagation in Neural Networks •Explaining gradient descent starts in many articles or tutorials with mountains. Imagine you are put on a mountain, not necessarily the top, by a helicopter at night or heavy fog. Let's further imagine that this mountain is on an island and you want to reach sea level. You have to go down, but you hardly see anything, maybe just a few metres.
- 35. 35 Backpropagation in Neural Networks •Your task is to find your way down, but you cannot see the path. You can use the method of gradient descent. This means that you are examining the steepness at your current position. You will proceed in the direction with the steepest descent. You take only a few steps and then you stop again to reorientate yourself. This means you are applying again the previously described procedure, i.e. you are looking for the steepest descend.
- 36. 36 Backpropagation in Neural Networks •This procedure is depicted in the following diagram in a two-dimensional space.
- 37.
- 38. 38 Backpropagation in Neural Networks •Going on like this you will arrive at a position, where there is no further descend. • Each directions goes upwards. You may have reached the deepest level - the global minimum -, but you might as well be stuck in a basin. If you start at the position on the right side of our image, everything works out fine, but from the leftside, you will be stuck in a local minimum. If you imagine now, - not very realistic - you are dropped many time at random places on this island, you will find ways downwards to sea level. This is what we actually do, when we train a neural network. •
- 39. 39 Machine Learning by usingPython Neural Network Lesson 2 By: Professor Lili Saghafi