DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
(ARTIFICIAL INTELLIGENCE & MACHINE LEARNING)
History of Deep Learning
Deep learning is based on the concept of artificial neural networks, or computational systems that
mimic the way the human brain functions. And so, our brief history of deep learning must start with those
neural networks.
1943: Warren McCulloch and Walter Pitts create a computational model for neural networks based on
mathematics and algorithms called threshold logic.
1958: Frank Rosenblatt creates the perceptron, an algorithm for pattern recognition based on a two-layer
computer neural network using simple addition and subtraction. He also proposed additional layers with
mathematical notations, but these wouldn’t be realised until 1975.
1980: Kunihiko Fukushima proposes the Neoconitron, a hierarchical, multilayered artificial neural
network that has been used for handwriting recognition and other pattern recognition problems.
1989: Scientists were able to create algorithms that used deep neural networks, but training times for the
systems were measured in days, making them impractical for real-world use.
1992: Juyang Weng publishes Cresceptron, a method for performing 3-D object recognition
automatically from cluttered scenes.
Mid-2000s: The term “deep learning” begins to gain popularity after a paper by Geoffrey Hinton and
Ruslan Salakhutdinov showed how a many-layered neural network could be pre-trained one layer at a
time.
2009: NIPS Workshop on Deep Learning for Speech Recognition discovers that with a large enough data
set, the neural networks don’t need pre-training, and the error rates drop significantly.
2012: Artificial pattern-recognition algorithms achieve human-level performance on certain tasks. And
Google’s deep learning algorithm discovers cats.
2014: Google buys UK artificial intelligence startup Deepmind for £400m
The Generative Adversarial Neural Network (GAN) was introduced in 2014. GAN was created by Ian
Goodfellow. With GAN, two neural networks play against each other in a game. The goal of the game is
for one network to imitate a photo, and trick its opponent into believing it is real. The opponent is, of
course, looking for flaws. The game is played until the near perfect photo tricks the opponent. GAN
provides a way to perfect a product (and has also begun being used by scammers).
Department of Computer Science & Engineering-(AI&ML) | APSIT
DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
(ARTIFICIAL INTELLIGENCE & MACHINE LEARNING)
2015: Facebook puts deep learning technology – called DeepFace – into operations to automatically tag
and identify Facebook users in photographs. Algorithms perform superior face recognition tasks using
deep networks that take into account 120 million parameters.
2016: Google DeepMind’s algorithm AlphaGo masters the art of the complex board game Go and beats
the professional go player Lee Sedol at a highly publicised tournament in Seoul.
The promise of deep learning is not that computers will start to think like humans. That’s a bit
like asking an apple to become an orange. Rather, it demonstrates that given a large enough data set, fast
enough processors, and a sophisticated enough algorithm, computers can begin to accomplish tasks that
used to be completely left in the realm of human perception — like recognising cat videos on the web
(and other, perhaps more useful purposes).
Banks and financial services are using deep learning to automate trading, reduce risk, detect
fraud, and provide AI/chatbot advice to investors. A report from the EIU (Economist Intelligence Unit)
suggests 86% of financial services are planning to increase their artificial intelligence investments by
2025.
Department of Computer Science & Engineering-(AI&ML) | APSIT
DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING
(ARTIFICIAL INTELLIGENCE & MACHINE LEARNING)
HOW CAN YOU STAY CURRENT?
1. arxiv-sanity is a machine learning (though not deep learning!) powered tool from Andrej Karpathy to
help you shift through the vast swaths of research that is produced daily. Makes it easy to find papers
relevant to your interests and even comes with a Netflix style recommendation engine.
2. The keras blog has a lot of great tutorials on how to implement state of the art models in keras.
3. The machine learning subreddit contains links to and discussions of the latest and greatest papers.
Like all online communities, it can occasionally devolve into unproductive infighting, but I still find
it to be tremendously useful.
4. The OpenAI blog - Mainly focused on GANs and reinforcement learning
5. Google research blog - All of Google research, but with a huge serving of deep learning since this is
one of their main areas of focus.
6. Subscribe to the talking machines podcast. Co-hosted by Ryan Adam and Katherine Gorman, talking
machines is one of my favorite podcasts. Now two full seasons (with more to come!) of interviews
with luminaries of the field and discussion of the latest and greatest papers. Highly recommended!
Department of Computer Science & Engineering-(AI&ML) | APSIT
