Adventures in Timespace - How Apache Flink Handles Time and Windows
Download as pptx, pdf3 likes1,269 views
The document explains how Apache Flink handles time and windowing in stream processing, emphasizing the differences between event time and processing time. It discusses the importance of windows for organizing infinite streaming data and introduces key concepts like out-of-order elements and watermarks. The content highlights the various use cases and practical implementation of event time handling through Flink's API.
Adventures in Timespace - How Apache Flink Handles Time and Windows
- 1. Notions of Time Aljoscha Krettek [email protected] @aljoscha How Apache Flink™ Handles Time and Windows
- 3. 3 Why Windows*? *not Microsoft Windows…
- 6. 6 In Streaming: Arriving data never stops!
- 7. 7 Solution: Put elements into buckets, these are called windows
- 8. 8 Window (5 min) Count #Hashtags Just saw #Trump on #CNN, super cool. :D Trump: 2394 Cheese: 12984 Money: 42
- 9. 9 What I didn’t mention • tweets have a timestamp, their event time • tweets from across the globe arrive with delay => tweets with different timestamps arrive out-of-order
- 10. Window (5 min) Count #Hashtags 12:34 (13.10.2015): Just saw #Trump on #CNN, super cool. :D Trump: 2394 Cheese: 12984 Money: 42 These arrive with 3 minutes slack Form windows based on processing time of the machine. Processing Time != Event Time 10
- 11. 11 Why do people use this? • easy to implement • low latency • this is what systems give you (Spark Streaming, Apex, Samza, Storm)* *not Google Cloud Dataflow
- 12. 12 Lets look at a more complex example.
- 13. 13 Window (5 min) Correlate Tweets and News something... These still have 3 min slack. These have 8 min slack. 12:33 (13.10.2015): Donald Trump speaks at Cheese conference. Processing Time != Event Time
- 14. Processing Time != Event Time => Mismatch in the timespace continuum
- 15. 15 Use cases • out-of-order elements • sources with delay • recovery/fault-tolerance • “catching up” with a stream Who does it? • Google Cloud Dataflow • Apache Flink
- 16. 16 How can we do this?
- 17. 17 We need a Global Clock that runs on event time instead of processing time.
- 18. 18 This is a source This is our window operator 1 0 0 0 0 1 2 1 2 1 1 This is the current event-time time 2 2 2 2 2 This is a watermark.
- 19. 19 Now, show me the API!
- 20. 20 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setStreamTimeCharacteristic(ProcessingTime); DataStream<Tweet> text = env.addSource(new TwitterSrc()); DataStream<Tuple2<String, Integer>> counts = text .flatMap(new ExtractHashtags()) .keyBy(“name”) .timeWindow(Time.of(5, MINUTES) .apply(new HashtagCounter()); Processing Time
- 21. 21 Event Time StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setStreamTimeCharacteristic(EventTime); DataStream<Tweet> text = env.addSource(new TwitterSrc()); text = text.assignTimestamps(new MyTimestampExtractor()); DataStream<Tuple2<String, Integer>> counts = text .flatMap(new ExtractHashtags()) .keyBy(“name”) .timeWindow(Time.of(5, MINUTES) .apply(new HashtagCounter());
- 22. 22 TL;DL* • stream data is infinite • windows are helpful • event-time != processing time • watermarks to the rescue • Flink can do it *too long, didn’t listen
- 24. 32-35 24-27 20-23 8-110-3 4-7 24 Tumbling Windows of 4 Seconds 123412 4 59 9 0 20 20 22212326323321 26 353642