Flink forward SF 2017: Elizabeth K. Joseph and Ravi Yadav - Flink meet DC/OS – Deploying Flink at Scale
0 likes505 views
The document discusses the deployment of Apache Flink at scale using DC/OS, which addresses various data service challenges such as resource management, task scheduling, and container orchestration. It outlines features of DC/OS 1.9, including workload management through pods and GPU scheduling, which enhance performance for machine learning applications. Additionally, it provides insights into interacting with DC/OS via a user interface, CLI, and its capabilities for application logging and metrics.
Flink forward SF 2017: Elizabeth K. Joseph and Ravi Yadav - Flink meet DC/OS – Deploying Flink at Scale
- 1. © 2017 Mesosphere, Inc. All Rights Reserved. 1 FlinkForward 2017 - San Francisco Flink meet DC/OS Deploying Apache Flink at Scale Elizabeth K. Joseph, @pleia2 Ravi Yadav, @RaaveYadav
- 2. © 2017 Mesosphere, Inc. All Rights Reserved. 2 Part 1 Introduction to Apache Mesos, Marathon, and DC/OS Part 2 Demonstration of demo data pipeline + Installing Flink on DC/OS Part 3 DC/OS 1.9 key features for data services and beyond Talk Outline
- 3. © 2017 Mesosphere, Inc. All Rights Reserved. 3 Apache Mesos: The datacenter kernel http://mesos.apache.org/
- 4. © 2017 Mesosphere, Inc. All Rights Reserved. 4 ● Mesos can’t run applications on its own. ● A Mesos framework is a distributed system that has a scheduler. ● Schedulers like Marathon start and keep your applications running. A bit like a distributed init system. ● Mesos mechanics are fair and HA ● Learn more at https://mesosphere.github.io/marat hon/ Marathon
- 5. © 2017 Mesosphere, Inc. All Rights Reserved. 5 Introducing DC/OS Solves common problems ● Resource management ● Task scheduling ● Container orchestration ● Self-healing infrastructure ● Logging and metrics ● Network management ● “Universe” of pre-configured apps (including Flink, Kafka…) ● Learn more and contribute at https://dcos.io/
- 6. © 2017 Mesosphere, Inc. All Rights Reserved. 6 DC/OS Architecture Overview Security & Governance Container Orchestration Monitoring & Operations User Interface & Command Line HDFS Jenkins Marathon Cassandra Flink Spark Docker Kafka MongoDB +30 more... DC/OS Services & Containers ANY INFRASTRUCTURE
- 7. © 2017 Mesosphere, Inc. All Rights Reserved. 7 Web-based GUI https://dcos.io/docs/lates t/usage/webinterface/ Interact with DC/OS (1/2)
- 8. © 2017 Mesosphere, Inc. All Rights Reserved. 8 Universe
- 9. © 2017 Mesosphere, Inc. All Rights Reserved. 9 CLI tool https://dcos.io/docs/latest/usage/cli/ API https://dcos.io/docs/latest/api/ Interact with DC/OS (2/2)
- 10. © 2017 Mesosphere, Inc. All Rights Reserved. 10 According to the December 2016 data Artisans-organized Apache Flink user survey just under 30% of respondents were running Flink on Apache Mesos https://dcos.io/blog/2017/apache-flink-on-dc-os-and-apache-mesos/ You may already be using Apache Mesos! Version 1.2 of Flink includes support for Apache Mesos and DC/OS, “it is now possible to run an highly available Flink cluster on Mesos” https://flink.apache.org/news/2017/02/06/release-1.2.0.html#run-flink-with-apache- mesos & https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/mesos.html Flink on Apache Mesos and DC/OS
- 11. © 2017 Mesosphere, Inc. All Rights Reserved. 11 DEMOSDemo data pipeline + Installing Flink on DC/OS
- 12. © 2017 Mesosphere, Inc. All Rights Reserved. DC/OS 1.9 - Data Services Ecosystem WORKLOADS ● Pods ● GPU based scheduling DATA SERVICES ECOSYSTEM OPERATIONS ● Remote Container Shell ● Unified Metrics ● Deployment failure analyzer ● Alluxio ● Couchbase ● Datastax DSE ● Elastic (ELK) ● Redis ● Apache Flink
- 13. © 2017 Mesosphere, Inc. All Rights Reserved. DC/OS 1.9 - Operations WORKLOADS ● Pods ● GPU based scheduling DATA SERVICES ECOSYSTEM OPERATIONS ● Remote Container Shell ● Unified Metrics ● Unified Logging ● Deployment Failure Debugging ● Upgrades & Configuration updates
- 14. © 2017 Mesosphere, Inc. All Rights Reserved. 14 ● Open encrypted, interactive, remote session to your containers ● Remotely execute commands for real time app troubleshooting ● Provide developers access to their own applications, not the entire host or cluster my-laptop$ dcos task exec my-task /bin/bash Starting /bin/bash in my-task ... Connecting to remote my-task … REMOTE CONTAINER SHELL DC/OS: OPERATIONS DC/OS 1.9
- 15. © 2017 Mesosphere, Inc. All Rights Reserved. 15 UNIFIED LOGGING DC/OS: OPERATIONS ● Access application, DC/OS and OS logs ● Easily troubleshoot applications with critical metadata such as container id and app id ● Integrate easily with existing logging systems DC/OS 1.9
- 16. © 2017 Mesosphere, Inc. All Rights Reserved. 16 UNIFIED METRICS DC/OS: OPERATIONS ● Single API for system, container and application metrics ● Metadata such as host id and container id are automatically added to assist in debugging ● Integrate easily with existing metrics systems DC/OS 1.9 Container
- 17. © 2017 Mesosphere, Inc. All Rights Reserved. 17 DEPLOYMENT FAILURE DEBUGGING DC/OS: OPERATIONS ● Understand why your application is not deploying ● Understand which nodes in the cluster can accommodate the role, constraints, cpu, mem, disk and port requirements for your app DC/OS 1.9
- 18. © 2017 Mesosphere, Inc. All Rights Reserved. UPGRADES AND CONFIG UPDATES DC/OS: OPERATIONS DC/OS 1.9 18 ● Generate new config for cluster nodes $ dcos_generate_config.sh --generate-node-upgrade-script <installed_cluster_version> ● Single command upgrade script for individual nodes $ curl -O <Node upgrade script URL> $ sudo bash ./dcos_node_upgrade.sh
- 19. © 2017 Mesosphere, Inc. All Rights Reserved. DC/OS 1.9 - Workloads WORKLOADS ● Pods ● GPU based scheduling DATA SERVICES ECOSYSTEM OPERATIONS ● Remote Container Shell ● Unified Logging ● Unified Metrics ● Deployment failure analyzer
- 20. © 2017 Mesosphere, Inc. All Rights Reserved. 20 ● Schedule, deploy and scale multiple containers on the same host(s) while sharing IP address and storage volumes ● All containers in a pod instance run as if they are running on a single host in pre-container world ● Useful for migrating legacy applications or building advanced micro services (side car containers) PODS DC/OS: WORKLOADS DC/OS 1.9
- 21. © 2017 Mesosphere, Inc. All Rights Reserved. 21 ● Traditional monolithic apps on VMs usually have support services such as log shipper, message queuing clients ● Many support services assume col-location on same host, and local-host access to networking and storage ● Pods simplify moving legacy monolithic apps to containers, reducing risk and accelerating migrations DC/OS 1.9 PODS: MIGRATING LEGACY APPS TO CONTAINERS DC/OS: WORKLOADS
- 22. © 2017 Mesosphere, Inc. All Rights Reserved. 22 ● Advanced Micro Services patterns require colocating containers together ● Support services include for example: ○ Logging or monitoring agents, ○ Backup tooling & Proxies ○ Data change watchers & Event publishers ● Pods simplify the building and maintenance of complex such microservices DC/OS 1.9 PODS: SUPPORT SERVICES (SIDE-CAR CONTAINERS) DC/OS: WORKLOADS
- 23. © 2017 Mesosphere, Inc. All Rights Reserved. GPU: WHY GPU? DC/OS: WORKLOADS DC/OS 1.9 ● GPUs are needed for many machine learning and deep learning applications ● GPUs are essential for real-time or near real time machine learning models ● GPUs deliver from 10X to 100X performance for some applications, resulting lower $$$/IOPS and more productivity to data science teams ● GPU applications include real time fraud detection, genome sequencing, cohort analysis and many others
- 24. © 2017 Mesosphere, Inc. All Rights Reserved. GPU BASED SCHEDULING DC/OS: WORKLOADS DC/OS 1.9 24 ● Test Locally with Nvidia-Docker, deploy to production with DC/OS ● Isolate GPU instances and schedule workloads just like CPU and memory, guaranteeing performance ● Efficiently Share GPU resources across data science team ● Simplify migrating machine learning models across from dev to production, and across clouds
- 25. © 2017 Mesosphere, Inc. All Rights Reserved. OTHER IMPROVEMENTS DC/OS DC/OS 1.9 25 ● Mesos 1.2 ● Marathon 1.4 ● Docker 1.12 and 1.13 (17.03-ce) support ● Centos 7.3 and CoreOS 1235.12.0 support ● Performance improvements across all networking features. ● CNI support for 3rd party CNI plugins. ● 100s of additional bugfixes and tests