Timing is Everything: Understanding Event-Time Processing in Flink SQL
0 likes168 views
The document provides an overview of Apache Flink's event-time processing model and its applications in Flink SQL, highlighting concepts like watermarks, window types, and the trade-offs between correctness and latency. It discusses how event time is crucial for timely analytics and monitoring and presents different window types such as tumble, hop, cumulative, and session. The document emphasizes that Flink SQL simplifies event time processing and facilitates accurate computations despite challenges like out-of-order events.
Timing is Everything: Understanding Event-Time Processing in Flink SQL
- 1. Timing is Everything Understanding Event-Time Processing in Flink SQL Sharon Xie，Flink Babysitter Founding Engineer @ Decodable
- 2. Agenda - Flink’s event time processing model - Use cases in Flink SQL
- 3. What is Apache Flink Stateful Computations over Data Streams ● Highly Scalable ● Exactly-once processing semantics ● Event time semantics and watermarks ● Layered APIs: Streaming SQL (easy to use) ↔ DataStream (expressive)
- 4. Context ● Flink streaming mode ● Flink SQL ● Current Flink version - 1.18 1.19
- 5. When is event time used? ● Monitoring and alerting ● Time-based compute or analytics
- 6. Event Immutable record containing the detail of something that happened at some point in time.
- 7. Time in Flink Event Time ● The time at which the event happened Processing Time ● The time at which the event is observed by Flink
- 8. Event time vs Processing Time ● Event time is < processing time ● The lag is arbitrary ● Events can be out-of-order
- 9. Challenges How do you know when all of the events are received for a particular window?
- 10. Watermark ● Measures the progress of event time ● Tracks the maximum event time seen ● Indicates the completeness of the event time
- 12. Create table sensors ( id bigint, `value` integer _time timestamp(3), watermark for _time as _time - interval '3' minutes ) WITH ( 'scan.watermark.emit.strategy'='on-event', ... ); Define Watermark
- 14. There is a window that ends at 1:05. When can the window close? Quiz
- 15. There is a window that ends at 1:05. When can the window close? Quiz - Answer
- 17. Idle source/partition ● If a partition is idle (no events), the watermark will not advance ● No result will be produced ● Solutions ○ Configure source idle timeout ■ set table.exec.source.idle-timeout = 1m ○ Balance the partitions
- 18. Implications ● Tradeoff between Correctness and Latency ● Latency ○ Results of a window is only seen after the window closes ● Correctness ○ Late arriving events are discarded after the window is closed
- 19. Correctness VS Latency In general: Alerting and monitoring: latency Timely analytics: correctness
- 20. But…can I have both? ● Yes! Flink can process & emit “updates” (changelog) ● No watermark is needed ● Downstream system must support “updates” ● It’s costly - need to store global state
- 21. Trade-offs
- 22. Quick Summary ● Timely response & analytics are based on event time ● Flink uses watermark to account for out-of-order events ● Watermark allows trade-off between accuracy and latency
- 23. Flink SQL Event-time Usage ● Windowed Aggregations ● Window join ● Temporal join
- 24. Windowing Put unbounded events into finite-sized temporal buckets, over which computation is applied.
- 25. Window Types ● Tumble / Fixed ● Hop / Sliding ● Cumulative ● Session
- 26. Window Types - Tumble/Fixed Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ ● Fixed window size ● No overlapping ● Each event belongs to exactly 1 window
- 27. Flink SQL (Window TVF) ● TVF - Table-Valued Function ● Returns a new relation with all columns of original stream and additional 3 columns: ○ window_start, window_end, window_time
- 28. Example - Tumble Window
- 29. Window Types - Hop/Slide Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ ● Fixed window size ● Overlaps when slide < window size
- 30. Window Types - Cumulative Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ ● Similar to tumble window, but with early firing at the defined interval ● Defined by max window size and window step
- 31. Window Types - Session 😃 Supported in Flink 1.19 ● A new window is created when the consecutive event time > session gap Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/#session
- 32. Window Join ● A window join adds the dimension of time into the join criteria themselves. ● Use case: compute click-through events
- 33. Example - tumble window join
- 34. Example - hop window join
- 35. Temporal Join ● Enrich a stream with the value of the joined record at the event time. ● Example: Continuously computing the price for each order based on the exchange rate happened when the order is placed
- 36. Example - temporal join Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins/#temporal-joins
- 38. Summary ● Event time is essential for timely response and analytics ● Watermark and windowing are the key concepts ● Flink SQL simplifies event time processing
- 41. Q&A @sharon_rxie