Sergey Karayev, profile picture
Uploaded bySergey Karayev
41,494 views

Infrastructure and Tooling - Full Stack Deep Learning

The document outlines the infrastructure and tooling required for full stack deep learning, emphasizing the importance of robust data management, compute resources, and effective software engineering practices. It details the various components involved in model development, deployment, and experimentation, notably highlighting hardware options like GPUs and cloud services. The document also discusses programming languages, IDEs, and tools, while addressing challenges such as versioning and testing in ML workflows.

Technology◦
Related topics:
Deep Learning•

Embed presentation

Downloaded 461 times
1 / 106
2 / 106
3 / 106
4 / 106
5 / 106
6 / 106
7 / 106
8 / 106
9 / 106
10 / 106
11 / 106
12 / 106
13 / 106
14 / 106
15 / 106
16 / 106
17 / 106
18 / 106
19 / 106
20 / 106
21 / 106
22 / 106
23 / 106
24 / 106
25 / 106
26 / 106
27 / 106
28 / 106
29 / 106
30 / 106
31 / 106
32 / 106
33 / 106
34 / 106
35 / 106
36 / 106
37 / 106
38 / 106
39 / 106
40 / 106
41 / 106
42 / 106
43 / 106
44 / 106
45 / 106
46 / 106
47 / 106
48 / 106
49 / 106
50 / 106
51 / 106
52 / 106
53 / 106
54 / 106
55 / 106
56 / 106
57 / 106
58 / 106
59 / 106
60 / 106
61 / 106
62 / 106
63 / 106
64 / 106
65 / 106
66 / 106
67 / 106
68 / 106
69 / 106
70 / 106
71 / 106
72 / 106
73 / 106
74 / 106
75 / 106
76 / 106
77 / 106
78 / 106
79 / 106
80 / 106
81 / 106
82 / 106
83 / 106
84 / 106
85 / 106
86 / 106
87 / 106
88 / 106
89 / 106
90 / 106
91 / 106
92 / 106
93 / 106
94 / 106
95 / 106
96 / 106
97 / 106
98 / 106
99 / 106
100 / 106
101 / 106
102 / 106
103 / 106
104 / 106
105 / 106
106 / 106
Full Stack Deep Learning Josh Tobin, Sergey Karayev, Pieter Abbeel Infrastructure & Tooling
Full Stack Deep Learning Monitor predictions and close data flywheel loop Test and deploy model Run experiments, storing results Provision compute resources Write and debug model code 2 Provide data Get optimal prediction system
 as scalable API or mobile app Dream Reality Aggregate, clean, store, label,
 and version data Infrastructure & Tooling - Overview
Full Stack Deep Learning • Provide some labeled input-output pairs • Press a button • A prediction system that gets 100% accuracy on your data distribution is live (as an infinitely-scalable API, or running embedded or mobile) 3 SE4ML: Software Engineering for Machine Learning (NIPS 2014 Infrastructure & Tooling - Overview
Full Stack Deep Learning 4Infrastructure & Tooling - Overview
Full Stack Deep Learning 5 Goal: add data, see model improve Andrej Karpathy at PyTorch Devcon 2019 - https://www.youtube.com/watch?v=oBklltKXtDE Infrastructure & Tooling - Overview
Full Stack Deep Learning “All-in-one” 6 Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Infrastructure & Tooling - Overview Web Versioning Processing CI / Testing ? Resource Management
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Overview
Full Stack Deep Learning Questions? 8
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Software Engineering
Full Stack Deep Learning Programming Language • Python, because of the libraries • Clear winner in scientific and data computing 10
Full Stack Deep Learning Editors 11 VS CodeVim PyCharmEmacs Jupyter Infrastructure & Tooling - Software Engineering
Full Stack Deep Learning Visual Studio Code • VS Code makes for a very nice Python experience 12 Built-in git staging and diffing Open whole projects remotely Peek documentation Lint code as your write Infrastructure & Tooling - Software Engineering
Full Stack Deep Learning Linters and Type Hints • Whatever code style rules can be codified, should be • Static analysis can catch some bugs • Static type checking both documents code and catches bugs • Will see in Lab 7 13Infrastructure & Tooling - Software Engineering
Full Stack Deep Learning Jupyter Notebooks • Notebooks have become fundamental to data science • Great as the "first draft" of a project • Jeremy Howard from fast.ai good to 
 learn from (course.fast.ai videos) • Difficult to make scalable, reproducible, well-tested • Counter-point: Netflix based all ML workflows on them 14Infrastructure & Tooling - Software Engineering https://medium.com/netflix-techblog/notebook-innovation-591ee3221233
Full Stack Deep Learning Problems with notebooks • Hard to version • Notebook "IDE" is primitive • Very hard to test • Out-of-order execution artifacts • Hard to run long or distributed tasks 15 https://towardsdatascience.com/5-reasons-why-jupyter-notebooks-suck-4dc201e27086 Infrastructure & Tooling - Software Engineering
Full Stack Deep Learning Streamlit • New, but great at fulfilling a common ML need: interactive applets • Decorate normal Python code • Smart data caching, quick re-rendering • In the works: sharing as easy as pushing a web app to Heroku 16 https://streamlit.io Infrastructure & Tooling - Software Engineering
Full Stack Deep Learning Questions? 17
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Compute
Full Stack Deep Learning 19 Development Training/Evaluation • Function • Writing code • Debugging models • Looking at results • Desiderata • Quickly compile models and run training • Nice-to-have: use GUI • Solutions • Desktop with 1-4 GPUs • Cloud instance with 1-4 GPUs • Function • Model architecture / hyperparam search • Training large models • Desiderata • Easy to launch experiments and review results • Solutions • Desktop with 4 GPUs • Private cluster of GPU machines • Cloud cluster of GPU instances or or Infrastructure & Tooling - Compute Compute needs
Full Stack Deep Learning Why compute matters 20 https://openai.com/blog/ai-and-compute/ Infrastructure & Tooling - Compute
Full Stack Deep Learning But creativity matters, too! 21 https://openai.com/blog/ai-and-compute/ Infrastructure & Tooling - Compute https://www.fast.ai/2018/04/30/dawnbench-fastai/
Full Stack Deep Learning So, or ? • GPU Basics • Cloud Options • On-prem Options • Analysis and Recommendations 22Infrastructure & Tooling - Compute
Full Stack Deep Learning GPU Basics • NVIDIA has been the only game in town • Google TPUs are the fastest current option (on GCP) • Intel Nervana NNPs and AMD could start making sense in the future 23Infrastructure & Tooling - Compute
Full Stack Deep Learning 24 Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500 GPU Comparison Table Infrastructure & Tooling - Compute https://www.microway.com/knowledge-center-articles/comparison-of-nvidia-geforce-gpus-and-nvidia-tesla-gpus/
Full Stack Deep Learning 25 GPU Comparison Table Infrastructure & Tooling - Compute • New NVIDIA architecture every year • Kepler —> Maxwell —> Pascal —> Volta -> Turing • Server version first, then “enthusiast”, then consumer. • For business, only supposed to use server cards. Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 26 GPU Comparison Table Infrastructure & Tooling - Compute • RAM: should fit meaningful batches of your model • Most important for recurrent models http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 27 GPU Comparison Table Infrastructure & Tooling - Compute • RAM: should fit meaningful batches of your model • Most important for recurrent models • 32bit vs Tensor TFlops • Tensor Cores are specifically for deep learning operations (mixed precision) • Good for convolutional/transformer models http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 28 GPU Comparison Table Infrastructure & Tooling - Compute • RAM: should fit meaningful batches of your model • Most important for recurrent models • 32bit vs Tensor TFlops • Tensor Cores are specifically for deep learning operations (mixed precision) • Good for convolutional/transformer models • Straight 16bit is a bit less good but still better than 32bit • roughly 2x flops and 1.5x memory http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 29 Kepler/Maxwell • 2-4x slower than Pascal/Volta • Hardware: don’t buy, too old • Cloud: K80's are cheap (providers are stuck with what they bought) Infrastructure & Tooling - Compute Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 30 Pascal/Volta • Hardware: 1080 Ti still good if buying used, especially for recurrent • Cloud: P100 is a mid-range option Infrastructure & Tooling - Compute Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 31 Turing • Preferred choice right now due to 16bit mixed precision support • Hardware: • 2080 Ti is ~1.3x as fast as 1080 Ti in 32bit, but ~2x faster in 16bit • Titan RTX is 10-20% faster yet. Titan V is just as good (but less RAM), if find used. • Cloud: V100 is the ultimate for speed http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ https://lambdalabs.com/blog/titan-rtx-tensorflow-benchmarks/ Infrastructure & Tooling - Compute Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
Full Stack Deep Learning 32 https://timdettmers.com/2019/04/03/which-gpu-for-deep-learning/ Infrastructure & Tooling - Compute
Full Stack Deep Learning Cloud Providers • Amazon Web Services, Google Cloud Platform, Microsoft Azure are the heavyweights. • Heavyweights are largely similar in function and price. • AWS most expensive. • GCP lets you connect GPUs to any instance, and has TPUs. • Azure reportedly has bad user experience • Startups are Paperspace, Lambda Labs 33Infrastructure & Tooling - Compute
Full Stack Deep Learning 34 Amazon Web Services Name GPU GPUs GPU RAM vCPU RAM On-demand Spot p3.2xlarge V100 1 16 8 61 $3.06 $1.05 p3.8xlarge V100 4 64 32 244 $12.24 $4.20 p3.16xlarge V100 8 128 64 488 $24.48 $9.64 p2.xlarge K80 1 die 12 4 61 $0.90 $0.43 p2.8xlarge K80 8 dies 96 32 488 $7.20 $3.40 p2.16xlarge K80 16 dies 192 64 768 $14.40 $6.80 Infrastructure & Tooling - Compute
Full Stack Deep Learning 35 Amazon Web Services Name GPU GPUs GPU RAM vCPU RAM On-demand Spot p3.2xlarge V100 1 16 8 61 $3.06 $0.95 p3.8xlarge V100 4 64 32 244 $12.24 $4.00 p3.16xlarge V100 8 128 64 488 $24.48 $7.30 p2.xlarge K80 1 die 12 4 61 $0.90 $0.27 p2.8xlarge K80 8 dies 96 32 488 $7.20 $2.16 p2.16xlarge K80 16 dies 192 64 768 $14.40 $4.32 • For a big (50-80%) discount, can get instances that can terminate at any time. • Makes sense for running hyperparam search experiments, but need infrastructure to handle failures. Infrastructure & Tooling - Compute
Full Stack Deep Learning 36 Google Cloud Platform GPU GPUs GPU RAM On-demand Spot V100 1-8 16 $2.48 $0.74 P100 1-4 64 $1.46 $0.43 K80 1-8 12 $0.45 $0.14 • GPUs can be attached to any instance. • In general, a little cheaper than AWS. • Also has “Tensor Processing Units” (TPUs), the fastest option today. Infrastructure & Tooling - Compute
Full Stack Deep Learning 37 Microsoft Azure Name GPU GPUs GPU RAM vCPU RAM On-demand `nc6 v3` V100 1 16 6 112 $3.06 `nc12 v3` V100 2 32 12 224 $6.12 `nc24 v3` V100 4 64 24 448 $12.24 `nc6 v2` P100 1 16 6 112 $2.07 `nc12 v2` P100 2 32 12 224 $4.14 `nc24 v2` P100 4 64 24 446 $8.28 `nc6` K80 1 24 4 56 $0.90 `nc12` K80 2 48 32 112 $7.20 `nc24` K80 4 96 64 224 $14.40 https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/ • Pretty much equivalent to AWS • Spot instances are not as well developed as in AWS or GCP. Infrastructure & Tooling - Compute
Full Stack Deep Learning On-prem Options • Build your own • Up to 4 GPUs is easy and quiet • Buy pre-built • Lambda Labs, NVIDIA, and builders like Supermicro, Cirrascale, etc. 38Infrastructure & Tooling - Compute
Full Stack Deep Learning Building your own • Quiet PC with 64GB RAM and 4x RTX 2080 Ti’s: $7000 • One day to build and set up • Going beyond 4 GPUs is painful • All you need to know: http:// timdettmers.com/2018/12/16/ deep-learning-hardware-guide/ 39Infrastructure & Tooling - Compute
Full Stack Deep Learning Pre-built: NVIDIA • NVIDIA DGX Station • 4x V100, 20-core CPU, 256GB RAM • $50,000 40Infrastructure & Tooling - Compute
Full Stack Deep Learning Pre-built: Lambda Labs 41 RTX 6000 ==Titan RTX with better cooling Infrastructure & Tooling - Compute ~20% more expensive
 than building yourself
Full Stack Deep Learning Pre-built: Lambda Labs 42Infrastructure & Tooling - Compute
Full Stack Deep Learning Cost Analysis • Let’s first compare on-prem and equivalent cloud machines • Then let’s also consider spot instances for experiment scaling 43Infrastructure & Tooling - Compute
Full Stack Deep Learning Quad PC vs Quad Cloud Verdict: not worth it. PC pays for itself in 5-10 weeks. 44 GPU Arch RAM Build Price Cloud Price Hours = Build 24/7 Weeks = Build 16/5 Weeks = Build 4x RTX 2080 Ti Volta 12 $10000.00 0 4x V100 Volta 16 $12.00 833 5 10 Full-time workload Work week load Infrastructure & Tooling - Compute
Full Stack Deep Learning Quad PC vs Quad Cloud 45 https://l7.curtisnorthcutt.com/build-pro-deep-learning-workstation
Full Stack Deep Learning Quad PC vs. Spot Instances How to think about it: cloud enables quicker experiments. 46 hours hours Length of trial in experiment (hours) 6 Number of trials in experiment 16 Total GPU hours for experiment 96 Cost of 4x RTX 2080 Ti machine $10,000.00 Time to run experiment on 4x machine 24 Time to run experiment on V100 spot instances 6 Cost of provisioning enough pre-emptible V100s $96.00 Number of experiments that equal cost of 4x machine 104 Infrastructure & Tooling - Compute
Full Stack Deep Learning Quad PC vs. Spot Instances But at a pretty steep price. 47 hours hours Length of trial in experiment (hours) 6 Number of trials in experiment 16 Total GPU hours for experiment 96 Cost of 4x RTX 2080 Ti machine $10,000.00 Time to run experiment on 4x machine 24 Time to run experiment on V100 spot instances 6 Cost of provisioning enough pre-emptible V100s $96.00 Number of experiments that equal cost of 4x machine 104 Infrastructure & Tooling - Compute
Full Stack Deep Learning In Practice • Even though cloud is expensive, it's hard to make on-prem scale past a certain point • Dev-ops (declarative infra, repeatable processe) definitely easier in the cloud 48Infrastructure & Tooling - Compute
Full Stack Deep Learning Recommendation for solo/startup • Development • Build or Buy a 4x Turing-architecture PC • Training/Evaluation • Use the same 4x GPU PC until architecture is dialed in • When running many experiments, either buy shared server machines or use cloud instances. 49Infrastructure & Tooling - Compute
Full Stack Deep Learning Recommendation for larger company • Development • Buy a 4x Turing-architecture PC per ML scientist • Or, let them use V100 instances • Training/Evaluation • Use cloud instances with proper provisioning and handling of failures 50Infrastructure & Tooling - Compute
Full Stack Deep Learning Questions? 51
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Resource Management
Full Stack Deep Learning 53 Resource Management • Function • Multiple people… • using multiple GPUs/machines… • running different environments • Goal • Easy to launch a batch of experiments, with proper dependencies and resource allocations • Solutions • Spreadsheet • Python scripts • SLURM • Docker + Kubernetes • Software specialized for ML use cases Infrastructure & Tooling - Resource Management
Full Stack Deep Learning Spreadsheets • People “reserve” what resources they need to use • Awful, but still surprisingly common 54Infrastructure & Tooling - Resource Management
Full Stack Deep Learning 55 • Problem we’re solving: allocate free resources to programs • Can be scripted pretty easily (see lab) • Even better, use old-school cluster job scheduler • Job defines necessary resources, gets queued Scripts or Infrastructure & Tooling - Resource Management
Full Stack Deep Learning 56 • Docker is a way to package up an entire dependency stack in a lighter-than-a-VM package • We will talk more about Docker in the Deployment lecture tomorrow, and use it in lab Docker + Kubernetes Infrastructure & Tooling - Resource Management
Full Stack Deep Learning 57 • Kubernetes is a way to run many docker containers on top of a cluster • (Is actually what you are doing the labs in: https://github.com/jupyterhub/zero- to-jupyterhub-k8s) Docker + Kubernetes Infrastructure & Tooling - Resource Management
Full Stack Deep Learning • Open source project from Google • Spawn and manage Jupyter notebooks • Manage multi-step ML workflows • Plug-ins for hyperparameter tuning, model deployment 58 https://github.com/Langhalsdino/Kubernetes-GPU-Guide Infrastructure & Tooling - Resource Management
Full Stack Deep Learning 59 Open Source project
 with paid features Infrastructure & Tooling - Resource Management
Full Stack Deep Learning Questions? 60
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Frameworks
Full Stack Deep Learning 62 Deep Learning Frameworks • Unless you have a good reason not to, use Tensorflow/Keras or PyTorch • Both are converging to the same ideal point: • easy development via define-by-run • multi-platform optimized execution graph • Today, most new projects use PyTorch • For most problems, fast.ai library worth starting with for immediate best practices Goodforproduction Good for development Tensorflow 2.0 + eager execution + TorchScript PyTorch 1.0 Infrastructure & Tooling - Frameworks 2013 2015 2015 2017 2018
Full Stack Deep Learning PyTorch dominates new development 63 https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/ Infrastructure & Tooling - Frameworks
Full Stack Deep Learning 64 Tensorflow/Keras still lead job posts Google searches (2018) Job posts (2018) https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a Infrastructure & Tooling - Frameworks
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Distributed Training
Full Stack Deep Learning Distributed Training • Using multiple GPUs and/or machines to train a single model. • More complex than simply running different experiments on different GPUs • Makes sense to reduce iteration time, especially on big datasets and large models 66Infrastructure & Tooling - Distributed Training
Full Stack Deep Learning Data Parallelism • If iteration time is too long, try training in data parallel regime • "For convolution, expect 1.9x/3.5x speedup for 2/4 GPUs.
 http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ 67 http://www.cs.cmu.edu/~pengtaox/papers/petuum_15.pdf https://lambdalabs.com/blog/titan-v-deep-learning-benchmarks/ This is average over
 different convnets Infrastructure & Tooling - Distributed Training
Full Stack Deep Learning Model Parallelism 68 http://www.cs.cmu.edu/~pengtaox/papers/petuum_15.pdf • Model parallelism is necessary when model does not fit on a single GPU • Introduces a lot of complexity and is usually not worth it • Better to buy the largest GPU you can Infrastructure & Tooling - Distributed Training
Full Stack Deep Learning Data-parallel Tensorflow • Can be quite easy: 69Infrastructure & Tooling - Distributed Training strategy = tf.distribute.MirroredStrategy() with strategy.scope(): model = tf.keras.Sequential([ tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D(), tf.keras.layers.Flatten(), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(10, activation='softmax') ]) model.compile(loss='sparse_categorical_crossentropy', optimizer=tf.keras.optimizers.Adam(),
Full Stack Deep Learning Data-parallel PyTorch • Can be quite easy: 70Infrastructure & Tooling - Distributed Training
Full Stack Deep Learning Data-parallel PyTorch • If want to spread over multiple machines, or make it work with model-parallel, gets more complicated 71Infrastructure & Tooling - Distributed Training In training code In consoles of the workers
Full Stack Deep Learning Ray • Ray is open-source project for effortless, stateful distributed computing in Python 72 https://ray.readthedocs.io/en/latest/walkthrough.html
Full Stack Deep Learning Horovod • Distributed training framework for Tensorflow, Keras, and PyTorch • Uses MPI (standard multi-process communication framework) instead of Tensorflow parameter servers • Could be an easier experience for multi-node training 73Infrastructure & Tooling - Distributed Training
Full Stack Deep Learning Questions? 74
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Experiment Management
Full Stack Deep Learning Experiment Management • Even running one experiment at a time, can lose track of which code, parameters, and dataset generated which trained model. • When running multiple experiments, problem is much worse. 76Infrastructure & Tooling - Experiment Management https://towardsdatascience.com/tracking-ml-experiments-using-mlflow-7910197091bb
Full Stack Deep Learning 77 Tensorboard • A fine solution for single experiments • Gets unwieldy to manage many experiments, and to properly store past work Infrastructure & Tooling - Experiment Management
Full Stack Deep Learning 78 Losswise Infrastructure & Tooling - Experiment Management
Full Stack Deep Learning 79 Comet.ml Infrastructure & Tooling - Experiment Management
Full Stack Deep Learning 80 Weights & Biases • What we use in Lab 3 Infrastructure & Tooling - Experiment Management
Full Stack Deep Learning 81 MLFlow tracking Infrastructure & Tooling - Experiment Management https://towardsdatascience.com/tracking-ml-experiments-using-mlflow-7910197091bb • Self-hosted solution from DataBricks
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Hyperparameter Tuning
Full Stack Deep Learning Hyperparameter Optimization • Useful to have software that helps you search over hyper parameter settings. • Could be as simple as being able to provide `—lr=(0.0001, 0.1) -- num_layers=[128, 256, 512]` to training script 83Infrastructure & Tooling - Hyperparameter Tuning
Full Stack Deep Learning 84Infrastructure & Tooling - Hyperparameter Tuning
Full Stack Deep Learning 85Infrastructure & Tooling - Hyperparameter Tuning
Full Stack Deep Learning Ray - Tune • "Choose among scalable SOTA algorithms such as Population Based Training (PBT), Vizier’s Median Stopping Rule, HyperBand/ASHA." • These redirect compute resources toward promising areas of search space 86Infrastructure & Tooling - Hyperparameter Tuning
Full Stack Deep Learning 87 Can also visually inspect hyperparam results Infrastructure & Tooling - Hyperparameter Tuning • We will also see a solution from W&B in Lab 4
Full Stack Deep Learning Questions? 88
Full Stack Deep Learning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - All-in-one
Full Stack Deep Learning All-in-one Solutions • Single system for everything • development (hosted notebook) • scaling experiments to many machines (sometimes even provisioning) • tracking experiments and versioning models • deploying models • monitoring performance 90Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 91Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 92Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 93Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 94Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 95Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 96 https://www.slideshare.net/AmazonWebServices/build-train-and-deploy-machine-learning-models-at-scale Infrastructure & Tooling - All-in-one 40% markup over corresponding EC2 instances
Full Stack Deep Learning 97Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 98Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 99Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 100Infrastructure & Tooling - All-in-one • HyperBand based hyperparameter tuning • In-house distributed training module
Full Stack Deep Learning Domino Data Lab 101 Provision compute Track experiments Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 102 Domino Data Lab Deploy REST API Monitor predictions Publish applets Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 103 Domino Data Lab Monitor spend All projects in one place Infrastructure & Tooling - All-in-one
Full Stack Deep Learning 104 Neptune Paperspace Gradient W&B Comet.ml Floyd Algorithmia Determined.ai Amazon SageMaker GC ML Engine Domino Data Lab Hardware GCP or agnostic Paperspace N/A N/A GCP N/A Agnostic AWS GCP Agnostic Resource Management Yes Yes No No Yes No Yes*
 (but not provisioning) Yes Yes Yes Hyperparam Optimization Yes No Yes Yes No No Yes Yes Yes Yes Storing Models Yes Yes Yes Yes Yes No Yes Yes Yes Yes Reviewing Experiments Yes Yes Yes Yes Yes No Yes No Yes Yes Deploying Models as REST API No No No No Yes Yes Yes Yes Yes Yes Monitoring No No No No No Yes No Yes Yes Yes Infrastructure & Tooling - All-in-one
Full Stack Deep Learning Questions? 105
Full Stack Deep Learning Thank you! 106

More Related Content

PDF
Lecture 11: ML Deployment & Monitoring (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Data Management - Full Stack Deep Learning
bySergey Karayev
 
PDF
Setting up Machine Learning Projects - Full Stack Deep Learning
bySergey Karayev
 
PDF
Machine Learning Teams - Full Stack Deep Learning
bySergey Karayev
 
PDF
Lecture 5: ML Projects (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Troubleshooting Deep Neural Networks - Full Stack Deep Learning
bySergey Karayev
 
PDF
Testing and Deployment - Full Stack Deep Learning
bySergey Karayev
 
PDF
Lecture 7: Troubleshooting Deep Neural Networks (Full Stack Deep Learning - S...
bySergey Karayev
 
Lecture 11: ML Deployment & Monitoring (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Data Management - Full Stack Deep Learning
bySergey Karayev
 
Setting up Machine Learning Projects - Full Stack Deep Learning
bySergey Karayev
 
Machine Learning Teams - Full Stack Deep Learning
bySergey Karayev
 
Lecture 5: ML Projects (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Troubleshooting Deep Neural Networks - Full Stack Deep Learning
bySergey Karayev
 
Testing and Deployment - Full Stack Deep Learning
bySergey Karayev
 
Lecture 7: Troubleshooting Deep Neural Networks (Full Stack Deep Learning - S...
bySergey Karayev
 

What's hot

PDF
COBOL to Apache Spark
byRakuten Group, Inc.
 
PDF
Machine Learning Infrastructure
bySigOpt
 
PDF
Deploying and Monitoring Heterogeneous Machine Learning Applications with Cli...
byDatabricks
 
PDF
Tensors Are All You Need: Faster Inference with Hummingbird
byDatabricks
 
PDF
Data Workflows for Machine Learning - Seattle DAML
byPaco Nathan
 
PPTX
Ai use cases
bySparsh Agarwal
 
PDF
AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeF...
byBill Liu
 
PDF
Drifting Away: Testing ML Models in Production
byDatabricks
 
PPTX
Jean-François Puget, Distinguished Engineer, Machine Learning and Optimizatio...
byMLconf
 
PDF
Lecture 4: Transformers (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Deep Learning for Natural Language Processing Using Apache Spark and TensorFl...
byDatabricks
 
PPTX
Predictive Model and Record Description with Segmented Sensitivity Analysis (...
byGreg Makowski
 
PDF
Agile Deep Learning
byDavid Murgatroyd
 
PDF
Machine learning model to production
byGeorg Heiler
 
PDF
Accelerate Your AI Today
byDESMOND YUEN
 
PDF
Data ops: Machine Learning in production
byStepan Pushkarev
 
PDF
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
byDatabricks
 
PDF
Managers guide to effective building of machine learning products
byGianmario Spacagna
 
PDF
AnalyticOps: Lessons Learned Moving Machine-Learning Algorithms to Production...
byRobert Grossman
 
PPTX
Production machine learning_infrastructure
byjoshwills
 
COBOL to Apache Spark
byRakuten Group, Inc.
 
Machine Learning Infrastructure
bySigOpt
 
Deploying and Monitoring Heterogeneous Machine Learning Applications with Cli...
byDatabricks
 
Tensors Are All You Need: Faster Inference with Hummingbird
byDatabricks
 
Data Workflows for Machine Learning - Seattle DAML
byPaco Nathan
 
Ai use cases
bySparsh Agarwal
 
AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeF...
byBill Liu
 
Drifting Away: Testing ML Models in Production
byDatabricks
 
Jean-François Puget, Distinguished Engineer, Machine Learning and Optimizatio...
byMLconf
 
Lecture 4: Transformers (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Deep Learning for Natural Language Processing Using Apache Spark and TensorFl...
byDatabricks
 
Predictive Model and Record Description with Segmented Sensitivity Analysis (...
byGreg Makowski
 
Agile Deep Learning
byDavid Murgatroyd
 
Machine learning model to production
byGeorg Heiler
 
Accelerate Your AI Today
byDESMOND YUEN
 
Data ops: Machine Learning in production
byStepan Pushkarev
 
Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...
byDatabricks
 
Managers guide to effective building of machine learning products
byGianmario Spacagna
 
AnalyticOps: Lessons Learned Moving Machine-Learning Algorithms to Production...
byRobert Grossman
 
Production machine learning_infrastructure
byjoshwills
 

Similar to Infrastructure and Tooling - Full Stack Deep Learning

PDF
Lecture 6: Infrastructure & Tooling (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PPTX
AI Hardware Landscape 2021
byGrigory Sapunov
 
PDF
FPGA Hardware Accelerator for Machine Learning
byDr. Swaminathan Kathirvel
 
PDF
Implementing AI: High Performace Architectures
byKTN
 
PDF
Tesla Accelerated Computing Platform
byinside-BigData.com
 
PDF
Enabling Artificial Intelligence - Alison B. Lowndes
byWithTheBest
 
PPTX
[DSC Europe 25] Dusan Gajic - Democratizing AI with Affordable HPC Infrastruc...
byDataScienceConferenc1
 
PDF
Deep Learning on the SaturnV Cluster
byinside-BigData.com
 
PDF
GTC 2017: Powering the AI Revolution
byNVIDIA
 
PDF
NVIDIA DGX User Group 1st Meet Up_30 Apr 2021.pdf
byMuhammadAbdullah311866
 
PDF
GTC 2016 Opening Keynote
byNVIDIA
 
PDF
Harnessing AI for the Benefit of All.
byAlison B. Lowndes
 
PDF
GPU Technology Conference 2014 Keynote
byNVIDIA
 
PDF
Dmitry Spodarets_Infrastructure for the work of data scientists
byFlyElephant
 
PDF
Infrastructure for the work of Data Scientists
byFlyElephant
 
PDF
infoShare AI Roadshow 2018 - Tomasz Kopacz (Microsoft) - jakie możliwości daj...
byInfoshare
 
PDF
Dl2 computing gpu
byArmando Vieira
 
PDF
AI, A New Computing Model
byNVIDIA Taiwan
 
PDF
Ai Forum at Computex 2017 - Keynote Slides by Jensen Huang
byNVIDIA Taiwan
 
PDF
Smart Data Slides: Emerging Hardware Choices for Modern AI Data Management
byDATAVERSITY
 
Lecture 6: Infrastructure & Tooling (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
AI Hardware Landscape 2021
byGrigory Sapunov
 
FPGA Hardware Accelerator for Machine Learning
byDr. Swaminathan Kathirvel
 
Implementing AI: High Performace Architectures
byKTN
 
Tesla Accelerated Computing Platform
byinside-BigData.com
 
Enabling Artificial Intelligence - Alison B. Lowndes
byWithTheBest
 
[DSC Europe 25] Dusan Gajic - Democratizing AI with Affordable HPC Infrastruc...
byDataScienceConferenc1
 
Deep Learning on the SaturnV Cluster
byinside-BigData.com
 
GTC 2017: Powering the AI Revolution
byNVIDIA
 
NVIDIA DGX User Group 1st Meet Up_30 Apr 2021.pdf
byMuhammadAbdullah311866
 
GTC 2016 Opening Keynote
byNVIDIA
 
Harnessing AI for the Benefit of All.
byAlison B. Lowndes
 
GPU Technology Conference 2014 Keynote
byNVIDIA
 
Dmitry Spodarets_Infrastructure for the work of data scientists
byFlyElephant
 
Infrastructure for the work of Data Scientists
byFlyElephant
 
infoShare AI Roadshow 2018 - Tomasz Kopacz (Microsoft) - jakie możliwości daj...
byInfoshare
 
Dl2 computing gpu
byArmando Vieira
 
AI, A New Computing Model
byNVIDIA Taiwan
 
Ai Forum at Computex 2017 - Keynote Slides by Jensen Huang
byNVIDIA Taiwan
 
Smart Data Slides: Emerging Hardware Choices for Modern AI Data Management
byDATAVERSITY
 

More from Sergey Karayev

PDF
Lecture 10: ML Testing & Explainability (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 8: Data Management (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring...
bySergey Karayev
 
PDF
AI Masterclass at ASU GSV 2019
bySergey Karayev
 
PDF
Research Directions - Full Stack Deep Learning
bySergey Karayev
 
PDF
Lecture 2.A: Convolutional Networks - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lecture 1: Deep Learning Fundamentals - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lecture 9: AI Ethics (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
KEY
Attentional Object Detection - introductory slides.
bySergey Karayev
 
PDF
Lab 1: Intro and Setup - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lecture 13: ML Teams (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
PDF
Lecture 3: RNNs - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
PDF
Lecture 12: Research Directions (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 10: ML Testing & Explainability (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 8: Data Management (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 2.B: Computer Vision Applications - Full Stack Deep Learning - Spring...
bySergey Karayev
 
AI Masterclass at ASU GSV 2019
bySergey Karayev
 
Research Directions - Full Stack Deep Learning
bySergey Karayev
 
Lecture 2.A: Convolutional Networks - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lecture 1: Deep Learning Fundamentals - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lecture 9: AI Ethics (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Attentional Object Detection - introductory slides.
bySergey Karayev
 
Lab 1: Intro and Setup - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lecture 13: ML Teams (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 
Lecture 3: RNNs - Full Stack Deep Learning - Spring 2021
bySergey Karayev
 
Lecture 12: Research Directions (Full Stack Deep Learning - Spring 2021)
bySergey Karayev
 

Recently uploaded

DOCX
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
PDF
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
PDF
Igniting the Future: Copilot trends, agentic transformation and product roadm...
byUni Systems S.M.S.A.
 
PDF
Session 1 - Solving Semi-Structured Documents with Document Understanding
byDianaGray10
 
PDF
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
PDF
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
PDF
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
PPTX
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
PPTX
The Landscape of Computer Software Development Companies
byYash Singh
 
PPT
software-security-intro in information security.ppt
byismaeelsahib032
 
PDF
API206-S: Transforming Supply Chains with Amazon Bedrock AgentCore - AWS re:I...
byChris Bingham
 
PPTX
THIS IS CYBER SECURITY NOTES USED IN CLASS ON VARIOUS TOPICS USED IN CYBERSEC...
byMelissaGblinwon
 
PPTX
The Future of IT Service Management AI Automation & Beyond.pptx
byChetu
 
PDF
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
PPTX
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
PDF
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
PPTX
cybercrime in Information security .pptx
byismaeelsahib032
 
PDF
Accelerating Responsible AI Adoption in Public Sector and Private Organizations.
byYasir Naveed Riaz
 
DOCX
Cloud Security, Serverless Security. Cybersecurity
byEdcelPacayraDuena
 
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
Igniting the Future: Copilot trends, agentic transformation and product roadm...
byUni Systems S.M.S.A.
 
Session 1 - Solving Semi-Structured Documents with Document Understanding
byDianaGray10
 
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
The Landscape of Computer Software Development Companies
byYash Singh
 
software-security-intro in information security.ppt
byismaeelsahib032
 
API206-S: Transforming Supply Chains with Amazon Bedrock AgentCore - AWS re:I...
byChris Bingham
 
THIS IS CYBER SECURITY NOTES USED IN CLASS ON VARIOUS TOPICS USED IN CYBERSEC...
byMelissaGblinwon
 
The Future of IT Service Management AI Automation & Beyond.pptx
byChetu
 
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
cybercrime in Information security .pptx
byismaeelsahib032
 
Accelerating Responsible AI Adoption in Public Sector and Private Organizations.
byYasir Naveed Riaz
 
Cloud Security, Serverless Security. Cybersecurity
byEdcelPacayraDuena
 

Infrastructure and Tooling - Full Stack Deep Learning

  • 1.
    Full Stack DeepLearning Josh Tobin, Sergey Karayev, Pieter Abbeel Infrastructure & Tooling
  • 2.
    Full Stack DeepLearning Monitor predictions and close data flywheel loop Test and deploy model Run experiments, storing results Provision compute resources Write and debug model code 2 Provide data Get optimal prediction system
 as scalable API or mobile app Dream Reality Aggregate, clean, store, label,
 and version data Infrastructure & Tooling - Overview
  • 3.
    Full Stack DeepLearning • Provide some labeled input-output pairs • Press a button • A prediction system that gets 100% accuracy on your data distribution is live (as an infinitely-scalable API, or running embedded or mobile) 3 SE4ML: Software Engineering for Machine Learning (NIPS 2014 Infrastructure & Tooling - Overview
  • 4.
    Full Stack DeepLearning 4Infrastructure & Tooling - Overview
  • 5.
    Full Stack DeepLearning 5 Goal: add data, see model improve Andrej Karpathy at PyTorch Devcon 2019 - https://www.youtube.com/watch?v=oBklltKXtDE Infrastructure & Tooling - Overview
  • 6.
    Full Stack DeepLearning “All-in-one” 6 Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Infrastructure & Tooling - Overview Web Versioning Processing CI / Testing ? Resource Management
  • 7.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Overview
  • 8.
    Full Stack DeepLearning Questions? 8
  • 9.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Software Engineering
  • 10.
    Full Stack DeepLearning Programming Language • Python, because of the libraries • Clear winner in scientific and data computing 10
  • 11.
    Full Stack DeepLearning Editors 11 VS CodeVim PyCharmEmacs Jupyter Infrastructure & Tooling - Software Engineering
  • 12.
    Full Stack DeepLearning Visual Studio Code • VS Code makes for a very nice Python experience 12 Built-in git staging and diffing Open whole projects remotely Peek documentation Lint code as your write Infrastructure & Tooling - Software Engineering
  • 13.
    Full Stack DeepLearning Linters and Type Hints • Whatever code style rules can be codified, should be • Static analysis can catch some bugs • Static type checking both documents code and catches bugs • Will see in Lab 7 13Infrastructure & Tooling - Software Engineering
  • 14.
    Full Stack DeepLearning Jupyter Notebooks • Notebooks have become fundamental to data science • Great as the "first draft" of a project • Jeremy Howard from fast.ai good to 
 learn from (course.fast.ai videos) • Difficult to make scalable, reproducible, well-tested • Counter-point: Netflix based all ML workflows on them 14Infrastructure & Tooling - Software Engineering https://medium.com/netflix-techblog/notebook-innovation-591ee3221233
  • 15.
    Full Stack DeepLearning Problems with notebooks • Hard to version • Notebook "IDE" is primitive • Very hard to test • Out-of-order execution artifacts • Hard to run long or distributed tasks 15 https://towardsdatascience.com/5-reasons-why-jupyter-notebooks-suck-4dc201e27086 Infrastructure & Tooling - Software Engineering
  • 16.
    Full Stack DeepLearning Streamlit • New, but great at fulfilling a common ML need: interactive applets • Decorate normal Python code • Smart data caching, quick re-rendering • In the works: sharing as easy as pushing a web app to Heroku 16 https://streamlit.io Infrastructure & Tooling - Software Engineering
  • 17.
    Full Stack DeepLearning Questions? 17
  • 18.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Compute
  • 19.
    Full Stack DeepLearning 19 Development Training/Evaluation • Function • Writing code • Debugging models • Looking at results • Desiderata • Quickly compile models and run training • Nice-to-have: use GUI • Solutions • Desktop with 1-4 GPUs • Cloud instance with 1-4 GPUs • Function • Model architecture / hyperparam search • Training large models • Desiderata • Easy to launch experiments and review results • Solutions • Desktop with 4 GPUs • Private cluster of GPU machines • Cloud cluster of GPU instances or or Infrastructure & Tooling - Compute Compute needs
  • 20.
    Full Stack DeepLearning Why compute matters 20 https://openai.com/blog/ai-and-compute/ Infrastructure & Tooling - Compute
  • 21.
    Full Stack DeepLearning But creativity matters, too! 21 https://openai.com/blog/ai-and-compute/ Infrastructure & Tooling - Compute https://www.fast.ai/2018/04/30/dawnbench-fastai/
  • 22.
    Full Stack DeepLearning So, or ? • GPU Basics • Cloud Options • On-prem Options • Analysis and Recommendations 22Infrastructure & Tooling - Compute
  • 23.
    Full Stack DeepLearning GPU Basics • NVIDIA has been the only game in town • Google TPUs are the fastest current option (on GCP) • Intel Nervana NNPs and AMD could start making sense in the future 23Infrastructure & Tooling - Compute
  • 24.
    Full Stack DeepLearning 24 Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500 GPU Comparison Table Infrastructure & Tooling - Compute https://www.microway.com/knowledge-center-articles/comparison-of-nvidia-geforce-gpus-and-nvidia-tesla-gpus/
  • 25.
    Full Stack DeepLearning 25 GPU Comparison Table Infrastructure & Tooling - Compute • New NVIDIA architecture every year • Kepler —> Maxwell —> Pascal —> Volta -> Turing • Server version first, then “enthusiast”, then consumer. • For business, only supposed to use server cards. Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 26.
    Full Stack DeepLearning 26 GPU Comparison Table Infrastructure & Tooling - Compute • RAM: should fit meaningful batches of your model • Most important for recurrent models http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 27.
    Full Stack DeepLearning 27 GPU Comparison Table Infrastructure & Tooling - Compute • RAM: should fit meaningful batches of your model • Most important for recurrent models • 32bit vs Tensor TFlops • Tensor Cores are specifically for deep learning operations (mixed precision) • Good for convolutional/transformer models http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 28.
    Full Stack DeepLearning 28 GPU Comparison Table Infrastructure & Tooling - Compute • RAM: should fit meaningful batches of your model • Most important for recurrent models • 32bit vs Tensor TFlops • Tensor Cores are specifically for deep learning operations (mixed precision) • Good for convolutional/transformer models • Straight 16bit is a bit less good but still better than 32bit • roughly 2x flops and 1.5x memory http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 29.
    Full Stack DeepLearning 29 Kepler/Maxwell • 2-4x slower than Pascal/Volta • Hardware: don’t buy, too old • Cloud: K80's are cheap (providers are stuck with what they bought) Infrastructure & Tooling - Compute Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 30.
    Full Stack DeepLearning 30 Pascal/Volta • Hardware: 1080 Ti still good if buying used, especially for recurrent • Cloud: P100 is a mid-range option Infrastructure & Tooling - Compute Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 31.
    Full Stack DeepLearning 31 Turing • Preferred choice right now due to 16bit mixed precision support • Hardware: • 2080 Ti is ~1.3x as fast as 1080 Ti in 32bit, but ~2x faster in 16bit • Titan RTX is 10-20% faster yet. Titan V is just as good (but less RAM), if find used. • Cloud: V100 is the ultimate for speed http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ https://lambdalabs.com/blog/titan-rtx-tensorflow-benchmarks/ Infrastructure & Tooling - Compute Card Release Arch Use-case RAM (Gb) 32bit TFlops Tensor TFlops 16bit Cost Cloud K80 2014H2 Kepler Server 24 5 N/A No used used AWS, GCP, MS Titan X 2015H1 Maxwell Enthusiast 12 6 N/A No used usedP100 2016H1 Pascal Server 16 10 N/A Yes used used GCP, MS 1080 Ti 2017H1 Pascal Consumer 11 13 N/A No used usedV100 2017H1 Volta Server 16 14 120 Yes $10000 AWS, GCP, MS Titan V 2017H2 Volta Enthusiast 12 14 110 Yes used used2080 Ti 2018H2 Turing Consumer 11 13 60 Yes $1000 Titan RTX 2018H2 Turing Enthusiast 24 16 130 Yes $2500 RTX 8000 2018H2 Turing Enthusiast 48 16 160 Yes $5500
  • 32.
    Full Stack DeepLearning 32 https://timdettmers.com/2019/04/03/which-gpu-for-deep-learning/ Infrastructure & Tooling - Compute
  • 33.
    Full Stack DeepLearning Cloud Providers • Amazon Web Services, Google Cloud Platform, Microsoft Azure are the heavyweights. • Heavyweights are largely similar in function and price. • AWS most expensive. • GCP lets you connect GPUs to any instance, and has TPUs. • Azure reportedly has bad user experience • Startups are Paperspace, Lambda Labs 33Infrastructure & Tooling - Compute
  • 34.
    Full Stack DeepLearning 34 Amazon Web Services Name GPU GPUs GPU RAM vCPU RAM On-demand Spot p3.2xlarge V100 1 16 8 61 $3.06 $1.05 p3.8xlarge V100 4 64 32 244 $12.24 $4.20 p3.16xlarge V100 8 128 64 488 $24.48 $9.64 p2.xlarge K80 1 die 12 4 61 $0.90 $0.43 p2.8xlarge K80 8 dies 96 32 488 $7.20 $3.40 p2.16xlarge K80 16 dies 192 64 768 $14.40 $6.80 Infrastructure & Tooling - Compute
  • 35.
    Full Stack DeepLearning 35 Amazon Web Services Name GPU GPUs GPU RAM vCPU RAM On-demand Spot p3.2xlarge V100 1 16 8 61 $3.06 $0.95 p3.8xlarge V100 4 64 32 244 $12.24 $4.00 p3.16xlarge V100 8 128 64 488 $24.48 $7.30 p2.xlarge K80 1 die 12 4 61 $0.90 $0.27 p2.8xlarge K80 8 dies 96 32 488 $7.20 $2.16 p2.16xlarge K80 16 dies 192 64 768 $14.40 $4.32 • For a big (50-80%) discount, can get instances that can terminate at any time. • Makes sense for running hyperparam search experiments, but need infrastructure to handle failures. Infrastructure & Tooling - Compute
  • 36.
    Full Stack DeepLearning 36 Google Cloud Platform GPU GPUs GPU RAM On-demand Spot V100 1-8 16 $2.48 $0.74 P100 1-4 64 $1.46 $0.43 K80 1-8 12 $0.45 $0.14 • GPUs can be attached to any instance. • In general, a little cheaper than AWS. • Also has “Tensor Processing Units” (TPUs), the fastest option today. Infrastructure & Tooling - Compute
  • 37.
    Full Stack DeepLearning 37 Microsoft Azure Name GPU GPUs GPU RAM vCPU RAM On-demand `nc6 v3` V100 1 16 6 112 $3.06 `nc12 v3` V100 2 32 12 224 $6.12 `nc24 v3` V100 4 64 24 448 $12.24 `nc6 v2` P100 1 16 6 112 $2.07 `nc12 v2` P100 2 32 12 224 $4.14 `nc24 v2` P100 4 64 24 446 $8.28 `nc6` K80 1 24 4 56 $0.90 `nc12` K80 2 48 32 112 $7.20 `nc24` K80 4 96 64 224 $14.40 https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/ • Pretty much equivalent to AWS • Spot instances are not as well developed as in AWS or GCP. Infrastructure & Tooling - Compute
  • 38.
    Full Stack DeepLearning On-prem Options • Build your own • Up to 4 GPUs is easy and quiet • Buy pre-built • Lambda Labs, NVIDIA, and builders like Supermicro, Cirrascale, etc. 38Infrastructure & Tooling - Compute
  • 39.
    Full Stack DeepLearning Building your own • Quiet PC with 64GB RAM and 4x RTX 2080 Ti’s: $7000 • One day to build and set up • Going beyond 4 GPUs is painful • All you need to know: http:// timdettmers.com/2018/12/16/ deep-learning-hardware-guide/ 39Infrastructure & Tooling - Compute
  • 40.
    Full Stack DeepLearning Pre-built: NVIDIA • NVIDIA DGX Station • 4x V100, 20-core CPU, 256GB RAM • $50,000 40Infrastructure & Tooling - Compute
  • 41.
    Full Stack DeepLearning Pre-built: Lambda Labs 41 RTX 6000 ==Titan RTX with better cooling Infrastructure & Tooling - Compute ~20% more expensive
 than building yourself
  • 42.
    Full Stack DeepLearning Pre-built: Lambda Labs 42Infrastructure & Tooling - Compute
  • 43.
    Full Stack DeepLearning Cost Analysis • Let’s first compare on-prem and equivalent cloud machines • Then let’s also consider spot instances for experiment scaling 43Infrastructure & Tooling - Compute
  • 44.
    Full Stack DeepLearning Quad PC vs Quad Cloud Verdict: not worth it. PC pays for itself in 5-10 weeks. 44 GPU Arch RAM Build Price Cloud Price Hours = Build 24/7 Weeks = Build 16/5 Weeks = Build 4x RTX 2080 Ti Volta 12 $10000.00 0 4x V100 Volta 16 $12.00 833 5 10 Full-time workload Work week load Infrastructure & Tooling - Compute
  • 45.
    Full Stack DeepLearning Quad PC vs Quad Cloud 45 https://l7.curtisnorthcutt.com/build-pro-deep-learning-workstation
  • 46.
    Full Stack DeepLearning Quad PC vs. Spot Instances How to think about it: cloud enables quicker experiments. 46 hours hours Length of trial in experiment (hours) 6 Number of trials in experiment 16 Total GPU hours for experiment 96 Cost of 4x RTX 2080 Ti machine $10,000.00 Time to run experiment on 4x machine 24 Time to run experiment on V100 spot instances 6 Cost of provisioning enough pre-emptible V100s $96.00 Number of experiments that equal cost of 4x machine 104 Infrastructure & Tooling - Compute
  • 47.
    Full Stack DeepLearning Quad PC vs. Spot Instances But at a pretty steep price. 47 hours hours Length of trial in experiment (hours) 6 Number of trials in experiment 16 Total GPU hours for experiment 96 Cost of 4x RTX 2080 Ti machine $10,000.00 Time to run experiment on 4x machine 24 Time to run experiment on V100 spot instances 6 Cost of provisioning enough pre-emptible V100s $96.00 Number of experiments that equal cost of 4x machine 104 Infrastructure & Tooling - Compute
  • 48.
    Full Stack DeepLearning In Practice • Even though cloud is expensive, it's hard to make on-prem scale past a certain point • Dev-ops (declarative infra, repeatable processe) definitely easier in the cloud 48Infrastructure & Tooling - Compute
  • 49.
    Full Stack DeepLearning Recommendation for solo/startup • Development • Build or Buy a 4x Turing-architecture PC • Training/Evaluation • Use the same 4x GPU PC until architecture is dialed in • When running many experiments, either buy shared server machines or use cloud instances. 49Infrastructure & Tooling - Compute
  • 50.
    Full Stack DeepLearning Recommendation for larger company • Development • Buy a 4x Turing-architecture PC per ML scientist • Or, let them use V100 instances • Training/Evaluation • Use cloud instances with proper provisioning and handling of failures 50Infrastructure & Tooling - Compute
  • 51.
    Full Stack DeepLearning Questions? 51
  • 52.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Resource Management
  • 53.
    Full Stack DeepLearning 53 Resource Management • Function • Multiple people… • using multiple GPUs/machines… • running different environments • Goal • Easy to launch a batch of experiments, with proper dependencies and resource allocations • Solutions • Spreadsheet • Python scripts • SLURM • Docker + Kubernetes • Software specialized for ML use cases Infrastructure & Tooling - Resource Management
  • 54.
    Full Stack DeepLearning Spreadsheets • People “reserve” what resources they need to use • Awful, but still surprisingly common 54Infrastructure & Tooling - Resource Management
  • 55.
    Full Stack DeepLearning 55 • Problem we’re solving: allocate free resources to programs • Can be scripted pretty easily (see lab) • Even better, use old-school cluster job scheduler • Job defines necessary resources, gets queued Scripts or Infrastructure & Tooling - Resource Management
  • 56.
    Full Stack DeepLearning 56 • Docker is a way to package up an entire dependency stack in a lighter-than-a-VM package • We will talk more about Docker in the Deployment lecture tomorrow, and use it in lab Docker + Kubernetes Infrastructure & Tooling - Resource Management
  • 57.
    Full Stack DeepLearning 57 • Kubernetes is a way to run many docker containers on top of a cluster • (Is actually what you are doing the labs in: https://github.com/jupyterhub/zero- to-jupyterhub-k8s) Docker + Kubernetes Infrastructure & Tooling - Resource Management
  • 58.
    Full Stack DeepLearning • Open source project from Google • Spawn and manage Jupyter notebooks • Manage multi-step ML workflows • Plug-ins for hyperparameter tuning, model deployment 58 https://github.com/Langhalsdino/Kubernetes-GPU-Guide Infrastructure & Tooling - Resource Management
  • 59.
    Full Stack DeepLearning 59 Open Source project
 with paid features Infrastructure & Tooling - Resource Management
  • 60.
    Full Stack DeepLearning Questions? 60
  • 61.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Frameworks
  • 62.
    Full Stack DeepLearning 62 Deep Learning Frameworks • Unless you have a good reason not to, use Tensorflow/Keras or PyTorch • Both are converging to the same ideal point: • easy development via define-by-run • multi-platform optimized execution graph • Today, most new projects use PyTorch • For most problems, fast.ai library worth starting with for immediate best practices Goodforproduction Good for development Tensorflow 2.0 + eager execution + TorchScript PyTorch 1.0 Infrastructure & Tooling - Frameworks 2013 2015 2015 2017 2018
  • 63.
    Full Stack DeepLearning PyTorch dominates new development 63 https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/ Infrastructure & Tooling - Frameworks
  • 64.
    Full Stack DeepLearning 64 Tensorflow/Keras still lead job posts Google searches (2018) Job posts (2018) https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a Infrastructure & Tooling - Frameworks
  • 65.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Distributed Training
  • 66.
    Full Stack DeepLearning Distributed Training • Using multiple GPUs and/or machines to train a single model. • More complex than simply running different experiments on different GPUs • Makes sense to reduce iteration time, especially on big datasets and large models 66Infrastructure & Tooling - Distributed Training
  • 67.
    Full Stack DeepLearning Data Parallelism • If iteration time is too long, try training in data parallel regime • "For convolution, expect 1.9x/3.5x speedup for 2/4 GPUs.
 http://timdettmers.com/2018/11/05/which-gpu-for-deep-learning/ 67 http://www.cs.cmu.edu/~pengtaox/papers/petuum_15.pdf https://lambdalabs.com/blog/titan-v-deep-learning-benchmarks/ This is average over
 different convnets Infrastructure & Tooling - Distributed Training
  • 68.
    Full Stack DeepLearning Model Parallelism 68 http://www.cs.cmu.edu/~pengtaox/papers/petuum_15.pdf • Model parallelism is necessary when model does not fit on a single GPU • Introduces a lot of complexity and is usually not worth it • Better to buy the largest GPU you can Infrastructure & Tooling - Distributed Training
  • 69.
    Full Stack DeepLearning Data-parallel Tensorflow • Can be quite easy: 69Infrastructure & Tooling - Distributed Training strategy = tf.distribute.MirroredStrategy() with strategy.scope(): model = tf.keras.Sequential([ tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D(), tf.keras.layers.Flatten(), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(10, activation='softmax') ]) model.compile(loss='sparse_categorical_crossentropy', optimizer=tf.keras.optimizers.Adam(),
  • 70.
    Full Stack DeepLearning Data-parallel PyTorch • Can be quite easy: 70Infrastructure & Tooling - Distributed Training
  • 71.
    Full Stack DeepLearning Data-parallel PyTorch • If want to spread over multiple machines, or make it work with model-parallel, gets more complicated 71Infrastructure & Tooling - Distributed Training In training code In consoles of the workers
  • 72.
    Full Stack DeepLearning Ray • Ray is open-source project for effortless, stateful distributed computing in Python 72 https://ray.readthedocs.io/en/latest/walkthrough.html
  • 73.
    Full Stack DeepLearning Horovod • Distributed training framework for Tensorflow, Keras, and PyTorch • Uses MPI (standard multi-process communication framework) instead of Tensorflow parameter servers • Could be an easier experience for multi-node training 73Infrastructure & Tooling - Distributed Training
  • 74.
    Full Stack DeepLearning Questions? 74
  • 75.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Experiment Management
  • 76.
    Full Stack DeepLearning Experiment Management • Even running one experiment at a time, can lose track of which code, parameters, and dataset generated which trained model. • When running multiple experiments, problem is much worse. 76Infrastructure & Tooling - Experiment Management https://towardsdatascience.com/tracking-ml-experiments-using-mlflow-7910197091bb
  • 77.
    Full Stack DeepLearning 77 Tensorboard • A fine solution for single experiments • Gets unwieldy to manage many experiments, and to properly store past work Infrastructure & Tooling - Experiment Management
  • 78.
    Full Stack DeepLearning 78 Losswise Infrastructure & Tooling - Experiment Management
  • 79.
    Full Stack DeepLearning 79 Comet.ml Infrastructure & Tooling - Experiment Management
  • 80.
    Full Stack DeepLearning 80 Weights & Biases • What we use in Lab 3 Infrastructure & Tooling - Experiment Management
  • 81.
    Full Stack DeepLearning 81 MLFlow tracking Infrastructure & Tooling - Experiment Management https://towardsdatascience.com/tracking-ml-experiments-using-mlflow-7910197091bb • Self-hosted solution from DataBricks
  • 82.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - Hyperparameter Tuning
  • 83.
    Full Stack DeepLearning Hyperparameter Optimization • Useful to have software that helps you search over hyper parameter settings. • Could be as simple as being able to provide `—lr=(0.0001, 0.1) -- num_layers=[128, 256, 512]` to training script 83Infrastructure & Tooling - Hyperparameter Tuning
  • 84.
    Full Stack DeepLearning 84Infrastructure & Tooling - Hyperparameter Tuning
  • 85.
    Full Stack DeepLearning 85Infrastructure & Tooling - Hyperparameter Tuning
  • 86.
    Full Stack DeepLearning Ray - Tune • "Choose among scalable SOTA algorithms such as Population Based Training (PBT), Vizier’s Median Stopping Rule, HyperBand/ASHA." • These redirect compute resources toward promising areas of search space 86Infrastructure & Tooling - Hyperparameter Tuning
  • 87.
    Full Stack DeepLearning 87 Can also visually inspect hyperparam results Infrastructure & Tooling - Hyperparameter Tuning • We will also see a solution from W&B in Lab 4
  • 88.
    Full Stack DeepLearning Questions? 88
  • 89.
    Full Stack DeepLearning “All-in-one” Deployment or or Development Training/Evaluation or Data or Experiment Management Labeling Interchange Storage Monitoring ? Hardware / Mobile Software Engineering Frameworks Database Hyperparameter TuningDistributed Training Web Versioning Processing CI / Testing ? Resource Management Infrastructure & Tooling - All-in-one
  • 90.
    Full Stack DeepLearning All-in-one Solutions • Single system for everything • development (hosted notebook) • scaling experiments to many machines (sometimes even provisioning) • tracking experiments and versioning models • deploying models • monitoring performance 90Infrastructure & Tooling - All-in-one
  • 91.
    Full Stack DeepLearning 91Infrastructure & Tooling - All-in-one
  • 92.
    Full Stack DeepLearning 92Infrastructure & Tooling - All-in-one
  • 93.
    Full Stack DeepLearning 93Infrastructure & Tooling - All-in-one
  • 94.
    Full Stack DeepLearning 94Infrastructure & Tooling - All-in-one
  • 95.
    Full Stack DeepLearning 95Infrastructure & Tooling - All-in-one
  • 96.
    Full Stack DeepLearning 96 https://www.slideshare.net/AmazonWebServices/build-train-and-deploy-machine-learning-models-at-scale Infrastructure & Tooling - All-in-one 40% markup over corresponding EC2 instances
  • 97.
    Full Stack DeepLearning 97Infrastructure & Tooling - All-in-one
  • 98.
    Full Stack DeepLearning 98Infrastructure & Tooling - All-in-one
  • 99.
    Full Stack DeepLearning 99Infrastructure & Tooling - All-in-one
  • 100.
    Full Stack DeepLearning 100Infrastructure & Tooling - All-in-one • HyperBand based hyperparameter tuning • In-house distributed training module
  • 101.
    Full Stack DeepLearning Domino Data Lab 101 Provision compute Track experiments Infrastructure & Tooling - All-in-one
  • 102.
    Full Stack DeepLearning 102 Domino Data Lab Deploy REST API Monitor predictions Publish applets Infrastructure & Tooling - All-in-one
  • 103.
    Full Stack DeepLearning 103 Domino Data Lab Monitor spend All projects in one place Infrastructure & Tooling - All-in-one
  • 104.
    Full Stack DeepLearning 104 Neptune Paperspace Gradient W&B Comet.ml Floyd Algorithmia Determined.ai Amazon SageMaker GC ML Engine Domino Data Lab Hardware GCP or agnostic Paperspace N/A N/A GCP N/A Agnostic AWS GCP Agnostic Resource Management Yes Yes No No Yes No Yes*
 (but not provisioning) Yes Yes Yes Hyperparam Optimization Yes No Yes Yes No No Yes Yes Yes Yes Storing Models Yes Yes Yes Yes Yes No Yes Yes Yes Yes Reviewing Experiments Yes Yes Yes Yes Yes No Yes No Yes Yes Deploying Models as REST API No No No No Yes Yes Yes Yes Yes Yes Monitoring No No No No No Yes No Yes Yes Yes Infrastructure & Tooling - All-in-one
  • 105.
    Full Stack DeepLearning Questions? 105
  • 106.
    Full Stack DeepLearning Thank you! 106