Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink Forward 2019 SF
11 likes•1,729 views
The document outlines the future of Apache Flink deployments, emphasizing advancements in containerization and Kubernetes integration. It discusses resource management, autoscaling, and the distinction between active and reactive execution modes for improved deployment efficiency. Additionally, it provides insights into the ongoing development of Flink's integration with Kubernetes and containerized environments to enhance operation ease.
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink Forward 2019 SF
- 1. © 2019 Ververica Till Rohrmann, Engineering Lead @stsffap Future of Apache Flink Deployments: Containers, Kubernetes and More
- 2. © 2019 Ververica2 Advancements of APIs & Supported Use Cases Batch Processing process static and historic data Data Stream Processing realtime results from data streams Event-driven Applications data-driven actions and services • Deployment & operations features did not keep up with APIs
- 3. © 2019 Ververica3 Flink as a Library • Deploying Flink applications should be as easy as starting a process • Application-oriented instead of cluster-oriented • Start multiple processes which form distributed execution • Removing the cluster out of the equation P1 P2 P3 P4 New process
- 4. © 2019 Ververica4 Resource Management & Autoscaling • Multiple short lived stages • Different resource requirements per stage • Efficient execution requires control over resources • Flink allocates actively resources • Long running, continuously processing operators • Constrained by external systems and SLAs • External system with holistic view over resources better suited for scaling decisions • Flink reacts to available resources Real-time & data-driven Applications Batch Analytics Continuous Processing
- 5. © 2019 Ververica5 Reactive vs. Active Execution Mode • Active mode – Flink is aware of underlying cluster framework – Flink allocate resources – E.g. existing YARN and Mesos integration • Reactive mode – Flink is oblivious to its runtime environment – External system allocates and releases resources – Flink scales with respect to available resources – Relevant for environments: Kubernetes, Docker, as a library
- 6. © 2019 Ververica6 Auto Scaling • Latency • Throughput • Connector signals • Available resources Scale up
- 7. © 2019 Ververica7 Containerized Deployments • Popular to deploy Flink in containerized environments • Why Flink on containers? – Self-contained runtime environment that includes application code, libraries, dependencies, and configuration files – Ease of operations by clean separation of concerns – Offers dynamic resource allocation • Reactive mode suits containerized deployments well • Demand for active mode to run batch workloads
- 8. © 2019 Ververica8 Containerized Deployments (cont.) • De-facto industry standard for container orchestration • Resource-oriented with declarative configuration – You tell K8s the desired state, and a background process asynchronously makes it happen – “3 replicas of this container should be kept running” – “a load balancer should exist, listening on port 443, backed by containers with this label” • Core resource types: – Pod: a group of one or more containers running on a node – Deployment: set of n identical pods Brief Introduction to Kubernetes
- 9. © 2019 Ververica Active & Reactive Mode
- 10. © 2019 Ververica10 Revamped Distributed Architecture (FLIP-6) – Better separation of concerns – RESTful API for client-cluster communication – Full resource elasticity Dispatcher Client ResourceManager ClusterManager JobManager TaskManager 1. Submit job 2. Start job 3. Request slots 4. Allocate resources 5. Start TaskManager 6. Execute job
- 11. © 2019 Ververica11 Active K8s Integration K8s deployment controller Client TaskManager JobManager K8sResMngr ApplicationMaster TaskManager (3) Submit job (1) Submit AM deployment (4) Start JM (5) Request slots (7) Start TM pod Dispatcher
- 12. © 2019 Ververica12 Reactive Mode • Relies on external system to start/release TaskManagers – Kubernetes Horizontal Pod Autoscaler – GCP Autoscaling – AWS Auto Scaling Group • Re-scale job if resources are added/removed • By definition works with all cluster managers Flink cluster JM TM TM ASG Start new TM if CPU% > threshold Register & offer slots Event rate over time
- 13. © 2019 Ververica Does Any of This Work? • Demo: Active K8s Integration & Reactive Container Mode
- 14. © 2019 Ververica14 Current State • Recommendations on how to deploy job and session cluster on K8s – Docker image build script – K8s resource configs • StandaloneJobClusterEntryPoint to start a (containerized) job cluster • Work in progress: – Active K8s integration • https://issues.apache.org/jira/browse/FLINK-9953 • Kudos to Chunhui-Shi • git clone -b native-k8s [email protected]:chunhui-shi/flink.git – Reactive Container Mode with automatic rescaling • https://issues.apache.org/jira/browse/FLINK-10407
- 15. © 2019 Ververica15 TL; DL • Reactive container mode – Support applications as first class citizen – Ease of operations • Active K8s integration coming soon • Join the community! – Subscribe to mailing lists – Participate in Flink development – Design the future
- 16. © 2019 Ververica Thank you! Questions?