Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015
9 likes•3,484 views
The document discusses fault tolerance and job recovery mechanisms in Apache Flink, highlighting the importance of implementing such features to prevent data loss and system downtime. It details concepts like checkpoints, barriers, and the different fault tolerance guarantees provided by Flink, enabling high availability and effective job processing. The overall message emphasizes that Flink ensures reliable data processing with low latency and high throughput through its advanced recovery strategies.
Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015
- 1. Fault Tolerance and Job Recovery in Apache Flink™ Till Rohrmann [email protected] @stsffap
- 2. 2
- 3. Better be safe than sorry § Failures will happen § EMC estimated $1.7 billion costs due to data loss and system downtime § Recovery will save you time and costs § Switch between algorithms § Live upgrade of your system 3
- 5. Fault tolerance guarantees § At most once • No guarantees at all § At least once • For many applications sufficient § Exactly once § Flink provides all guarantees 5
- 6. Checkpoints § Consistent snapshots of distributed data stream and operator state 6
- 7. Barriers § Markers for checkpoints § Injected in the data flow 7
- 8. 8 § Alignment for multi-input operators
- 9. Operator State § Stateless operators § System state § User defined state 9 ds.filter(_ != 0) ds.keyBy(0).window(TumblingTimeWindows.of(5, TimeUnit.SECONDS)) public class CounterSum implements RichReduceFunction<Long> { private OperatorState<Long> counter; @Override public Long reduce(Long v1, Long v2) throws Exception { counter.update(counter.value() + 1); return v1 + v2; } @Override public void open(Configuration config) { counter = getRuntimeContext().getOperatorState(“counter”, 0L, false); } }
- 10. 10
- 11. 11
- 12. 12
- 13. 13
- 14. Advantages § Separation of app logic from recovery • Checkpointing interval is just a config parameter § High throughput • Controllable checkpointing overhead § Low impact on latency 14
- 15. 15
- 18. With high availability 18 JobManager TaskManager Stand-by JobManager Apache Zookeeper™ KEEP GOING
- 20. Persisting jobs 20 JobManager Client TaskManagers Apache Zookeeper™ 1. Submit job 2. Persist execuAon graph
- 21. Persisting jobs 21 JobManager Client TaskManagers Apache Zookeeper™ 1. Submit job 2. Persist execuAon graph 3. Write handle to ZooKeeper
- 22. Persisting jobs 22 JobManager Client TaskManagers Apache Zookeeper™ 1. Submit job 2. Persist execuAon graph 3. Write handle to ZooKeeper 4. Deploy tasks
- 24. Handling checkpoints 24 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM
- 25. Handling checkpoints 25 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM 4. Create global checkpoint
- 26. Handling checkpoints 26 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM 4. Create global checkpoint 5. Persist global checkpoint
- 27. Handling checkpoints 27 JobManager Client TaskManagers Apache Zookeeper™ 1. Take snapshots 2. Persist snapshots 3. Send handles to JM 4. Create global checkpoint 5. Persist global checkpoint 6. Write handle to ZooKeeper
- 28. Conclusion 28
- 29. 29
- 30. 30
- 31. TL;DL § Job recovery mechanism with low latency and high throughput § Exactly one processing semantics § No single point of failure è Flink will always keep processing your data 31