More Related Content
Similar to AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeFlow, Airflow, and MLflow (20)
AISF19 - Building Scalable, Kubernetes-Native ML/AI Pipelines with TFX, KubeFlow, Airflow, and MLflow
- 1. Building Modern ML/AI Pipelines With the Latest Open Source Technologies KubeFlow + TFX + Airflow + MLflow Chris Fregly, Founder @ PipelineAI (Note: Slides Available on Twitter @cfregly)
- 2. Who am I? Founder @ PipelineAI Real-time Machine Learning and AI in Production Databricks Machine Learning Engineer O’Reilly Author “High Performance TensorFlow in Production” Meetup Organizer Advanced Spark and TensorFlow Meetup (SF) Netflix Personalization Engineer
- 3. Silicon Valley Venture Capital + Open Source 4,000 Stars = $6,000,000 Seed $1,500 per GitHub Star (According to one prominent VC on Sand Hill Road)
- 4. Popular Open Source ML Frameworks TensorFlow / Keras (Google) PyTorch (Facebook) MXNet (Amazon) July 2019 Survey
- 5. What is an ML Pipeline? System 6 System 5System 4 Training At Scale System 3 System 1 Data Ingestion Data Analysis Data Transform Data Validation System 2 Build Model Model Validation Serving Logging Monitoring Roll-out Data Splitting Ad-Hoc Training Distributed Training Configure
- 6. Complexity Brings Technical Debt https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
- 7. What is the Problem? Infrastructure Complexity Consistency Between Workflows and Tools Fragmentation One-Shot “Train and Forget” vs. Iterative
- 8. ML Infrastructure Docker Unit of deployment for model training and model serving 1 Docker image per batch job 1 Docker image per model server Kubernetes Manages and orchestrates Docker containers The new “Linux” of the data center First-class support for batch and cron jobs Current State: Fun for DevOps, Clunky for Data Scientists July 2019 Survey
- 9. Characteristics of a Modern ML Pipeline Experiment Tracking Compare different models or model hyper-parameters Hyper Parameter Optimization (HPO) Find the best hyper-parameters for a given model Distributed, Parallel Execution Perform independent pipeline components in parallel Metadata, Experiment Reproduce-ability, Data Versioning Back-in-time analysis of models and hyper-params on exact dataset Detect Model Bias / Decay / Drift Over time, models become crusty and irrelevant Feature Store Catalog of pre-engineered machine learning features across the entire organization July 2019 Survey
- 10. Experiment Tracking Compare Different Models or Model Hyper-Parameters Send all experiment results to a tracking server for comparison Offline Experiment Tracking is the Most Common Batch training jobs Online Experiment Tracking is an Emerging Field Compare model performance in live production A/B tests Multi-armed bandit tests
- 12. Hyper-Parameter Optimization (HPO) Search for Best Hyper-Parameters Naturally Parallelizable and Distributed Model Parameters != Model Hyper-Parameters Parameters: What the model learns. è Hyper-Parameters: How the model learns. è ie. Neural net architecture, depth of decision tree, learning rate of gradient descent
- 13. Demo: Hyper-Parameter Optimization (HPO) Parallel Coordinate Plot
- 14. Distributed, Parallel Execution Distributed, Parallel Execution Perform independent pipeline components in parallel Reduce/aggregate when complete TensorFlow Supports tf.distribute.HierarchicalCopyAllReduce
- 15. Demo: Distributed, Parallel Execution
- 16. Detect Model Bias / Decay / Drift
- 17. Metadata, Reproduce-ability, Data Versioning Back-in-time Analysis of Model Experiments, Hyper-Parameters, and Datasets* (*Requires Data to be Versioned for True Reproduce-ability) MetadataStore Saves All Inputs/Outputs of all ML Pipeline Components Individual Component
- 18. Feature Store Store and Share Results of Feature-Engineering in Common Repository Discover Pre-Engineered Features from Others in the Organization Examples of Feature Engineering: Remove data with `null` or `0` column Cross age and income to create a new feature Gender-balanced samples Current State: Multiple Startups are Actively Working on This Problem
- 19. Popular ML Pipeline Tools KubeFlow Kubernetes + TensorFlow Uses ModelDB open source project for experiment tracking and hyper-parameter tuning Airflow Initial focus is data engineering, emerging for feature engineering and ML / AI pipelines MLflow From-scratch implementation focused on experiment tracking & hyper-parameter tuning TensorFlow Extended (TFX) Set of TensorFlow-specific libraries to be combined with KubeFlow, Airflow, or Mlflow Pachyderm Initial focus is data versioning, emerging for ML / AI pipelines
- 20. KubeFlow Created by Google for “Kubernetes + TensorFlow” Many Existing OSS Projects Cobbled Together into 1 Tool Attempts to Create a Standard Language for ML Pipeline Definition Currently 0.6 Release Clunky For Now, but Getting Better
- 22. Airflow Created by Airbnb Originally Developed for Data Engineering Re-Purposed for Feature Engineering and ML Pipelines
- 24. MLflow
- 25. Demo: MLflow Experiment Tracking
- 26. TensorFlow Extended (TFX) Feature Load Feature Analyze Feature Transform Model Train Model Evalute Model Deploy Reproduce Training
- 27. Demo: TensorFlow Extended (TFX) Libraries Feature Load Feature Analyze Feature Transform Model Train Model Evalute Model Deploy Reproduce Training
- 28. Summary ML Pipelines are a Hot Topic! Multiple Tools and Multiple Standards Emerging Multiple Companies including Google, Facebook, Pinterest, Airbnb, Databricks, etc… We’re Still a Few Years from Mainstream Online Experiment Tracking and Optimization Suggestion: Stay Simple and Use Only What You Need
- 29. Thank You! Chris Fregly, Founder @ PipelineAI Free Community Edition https://community.pipeline.ai Contact Me [email protected] @cfregly