Download as PDF, PPTX
Uploaded byTill Rohrmann
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForward 2016 09/12/16)
This document discusses dynamic scaling in Apache Flink. It describes Flink's approach to dynamically scaling stateful jobs to adapt to changing workloads. Key points include: repartitioning of keyed and non-keyed state when scaling workers, supporting manual rescaling through savepoints and restarts currently, and future work on scaling operators without restarts and implementing automatic scaling policies.
More Related Content
![[215]네이버콘텐츠통계서비스소개 김기영](https://cdn.slidesharecdn.com/ss_thumbnails/215-161025030904-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForward 2016 09/12/16)
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForward 2016 09/12/16)
- 1. Till Rohrmann [email protected] @stsffap Dynamic Scaling:How Apache Flink® Adapts to Changing Workloads
- 2. Changing Workloads AndSLAs 2
- 3.
- 4. What Is ThisTalk About? § Flink’s approach to dynamic scaling § Current state with demo § Outlook on next development steps 4
- 5.
- 6. Basic Idea 6 • Spreadwork across more workers to decrease workload
- 7. Scaling Stateless Jobs 7 ScaleUp Scale Down Source Mapper Sink • Scale up: Deploy new tasks • Scale down: Cancel running tasks
- 8. Scaling Stateful Jobs 8 ? •Problem: Which state to assign to new task?
- 9. State in ApacheFlink 9
- 10. Keyed vs. Non-keyedState 10 • State bound to a key • E.g. Keyed UDF and window state • State bound to a subtask • E.g. Source state Keyed Non-keyed
- 11. Repartitioning Keyed State §Similar to consistent hashing § Split key space into key groups § Assign key groups to tasks 11 Key space Key group #1 Key group #2 Key group #3Key group #4
- 12. Repartitioning Keyed Statecontd. § Rescaling changes key group assignment § Maximum parallelism defined by #key groups 12
- 13. Repartitioning Non-keyed state §User defined merge and split functions • Most general approach § Breaking non-keyed state up into finer granularity • State has to contain multiple entries • Automatic repartitioning wrt granularity 13 #1 #2 #3
- 14. Repartitioning Non-keyed Statecontd. § Non-keyed state entries gathered at the job manager § Repartitioning schemes • Repartition & send • Union & broadcast 14
- 15. Example: Kafka Source 15 partitionId:1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 • Store offset for each partition • Individual entries are repartitionable partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27
- 16. Rescaling: Why isThat so Hard? § Handling of state § Repartitioning of keyed & non-keyed state § Unique among open source stream processors, afaik 16
- 17.
- 18. Demo Topology 18 Kafka SourceCounter KeyBy
- 19. Current State andnext Steps 19
- 20. Current State § Manualrescaling 1. Take savepoint 2. Stop the job 3. Restart job with adjusted parallelism and savepoint 20
- 21. Next Steps § Integratesavepoint with stop signal § Rescaling individual operators w/o restart § Dynamic container de-/allocation • “Running Flink Everywhere” by Stephan Ewen, 16:45 at Kesselhaus 21
- 22. Auto Scaling Policies 22 •Latency • Throughput • Resource utilization • Kubernetes on GCE, EC2 and Mesos (marathon- autoscale) already support auto-scaling
- 23. Conclusion § Scaling ofkeyed and non-keyed state § Flink supports manual rescaling with restart (WIP branch: https://github.com/tillrohrmann/flink/tree/partitionable-op-state) § Future versions might support scaling on the fly and automatic rescaling policies 23