Deep_Learning_Route
From Self-Driving Cars to Alpha Go to Language Translation, Deep Learning seems to be everywhere nowadays. While the debate whether the hype is justified or not continues, Deep Learning has seen a rapid surge of interest across academia and industry over the past years.
With so much attention on the topic, more and more information has been recently published, from various MOOCs to books to YouTube Channels. With such a vast amount of resources at hand, there has never been a better time to learn Deep Learning.
Yet a side effect of such an influx of readily available material is choice overload. With thousands and thousands of resources, which are the ones worth looking at?
Inspired by Haseeb Qureshi's excellent Guide on Learning Blockchain Development, this is my try to share the resources that I have found throughout my journey. As I am planning on continuously updating this guide with updated and better information, I deemed this work in progress an incomplete guide.
As of now, all the resources in this guide are all free thanks to their authors.
The link to my original blog post can be found here.
Target Audience
- Anybody who wants to dive deeper into the topic, seeks a career in this field or aspires to gain a theoretical understanding of Deep Learning
Goals of this Guide
- Give an overview and a sense of direction within the ocean of resources
- Give a clear and useful path towards learning Deep Learning Theory and Development
- Give some practical tips along the way on how to maximize your learning experience
Outline
This guide is structured in the following way:
- Phase 1 : Prerequisites
- Phase 2 : Deep Learning Fundamentals
- Phase 3 : Create Something
- Phase 4 : Dive Deeper
- Phase X : Keep Learning
How long your learning will take depends on numerous factors such as your dedication, background and time-commitment. And depending on your background and the things you want to learn, feel free to skip to any part of this guide. The linear progression I outline below is just the path I found to be useful for myself.
Phase 1: Pre-Requisites
Let me be clear from the beginning. The prerequisites you need depend on the objectives you intend to pursue. The foundations you need to conduct research in Deep Learning will differ from the things you need to become a Practitioner (both are of course not mutually exclusive).
Whether you don't have any knowledge of coding yet or you are already an expert in R, I would still recommend acquiring a working knowledge of Python as most of the resources on Deep Learning out there will require you to know Python.
Coding
While Codecademy is a great way to start coding from the beginning, MIT's 6.0001 lectures are an incredible introduction to the world of computer science. So, is CS50, Harvard's infamous CS intro course, but CS50 has less of a focus on Python. For people who prefer reading, the interactive online book How To Think Like A Computer Scientist is the way to go.
Math
If you simply want to apply Deep Learning techniques to a problem you face or gain a high-level understanding of Deep Learning, it is not necessary to know its mathematical underpinnings. But, in my experience, it has been significantly easier to understand and even more rewarding to use Deep Learning frameworks after getting familiar with its theoretical foundations. For such intentions, the basics of Calculus, Linear Algebra and Statistics are extremely useful. Fortunately, there are plenty of great Math resources online.
Here are the most important concepts you should know:
- Multivariable Calculus
- Differentiation
- Chain Rule
- Partial Derivatives
- Linear Algebra
- Definition of Vectors & Matrices
- Matrix Operations and Transformations: Addition, Subtraction, Multiplication, Transpose, Inverse
- Statistics & Probability
- Basic ideas like mean and standard deviation
- Distributions
- Sampling
- Bayes Theorem
That being said, it is also possible to learn these concepts concurrently with Phase 2, looking up the Math whenever you need it.
If you want to dive right into the Matrix Calculus that is used in Deep Learning, take a look at The Matrix Calculus You Need For Deep Learning by Terence Parr and Jeremy Howard.
For lectures, Gilbert Strang’s recorded videos of his course MIT 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning (2018) are on YouTube and covers Linear Algebra, Probability and Optimization in the context of Deep Learning.
Calculus
For Calculus, I would choose between the MIT OCW Lectures, Prof. Leonard's Lectures and Khan Academy. The MIT Lectures are great for people who are comfortable with Math and seek a fast-paced yet rigorous introduction to Calculus. Prof. Leonard Lectures are perfect for anybody who is not too familiar with Math as he takes the time to explain everything in a very understandable manner. Lastly, I would recommend Khan Academy for people who just need a refresher or want to get an overview as fast as possible.
Linear Algebra
For Linear Algebra, I really enjoyed Professor Strang's Lecture Series and its accompanying book on Linear Algebra (MIT OCW). If you are interested in spending more time on Linear Algebra, I would recommend the MIT lectures, but if you just want to learn the basics quickly or get a refresher, Khan Academy is perfect for that.
For a more hands-on coding approach, check out Rachel Thomas' (from fast.ai) Computational Linear Algebra Course.
At last, this review from Stanford's CS229 course offers a nice reference you can always come back to.
For both Calculus and Linear Algebra, 3Blue1Brown's Essence of Calculus and Linear Algebra series are beautiful complementary materials to gain a more intuitive and visual understanding of the subject.