Jamie Grier - Robust Stream Processing with Apache Flink

Jamie Grier - Robust Stream Processing with Apache Flink

• 1.
  Robust Stream Processing with Apache Flink JamieGrier @jamiegrier [email protected]
• 2.
  Who am I? •Director of Applications Engineering at data Artisans • Previously working on streaming computation at Twitter, Gnip and Boulder Imaging • Involved in various kinds of stream processing for about a decade • High-speed video, social media streaming, general frameworks for stream processing
• 3.
  Overview • What isApache Flink? • What is Stateful Stream Processing? • Windowed computation over streams • Robust Time Handling (Event Time vs Processing Time) • Robust Failure Handling • Robust Planned Downtime Handling • Robust Reprocessing
• 4.
  What is Apache Flink? ApacheFlink is an open source platform for distributed stream and batch data processing.
• 5.
  What is Apache Flink?
• 6.
  Stream Processing Your Code Data StreamData Stream
• 7.
  Stateful Stream Processing Your Code Data StreamData Stream State
• 8.
  More Complex Example Kafka Files Rabb itMQ Filter Map Join / Sum Influx DB C*
• 9.
  Distributed and Parallel Deployment Kafka Files Rabb it MQ Filter Parse Join/ Sum Influx DB C*
• 10.
  Robust Stream Processing withApache Flink
• 11.
  Code Example!
• 12.
  Windowing
• 13.
  Processing Time vs Event Time
• 14.
  Windowing in Processing Time 01 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Processing Time Event Time
• 15.
  Windowing in Event Time 01 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9 Event Time
• 16.
  Processing Time =Errors!
• 17.
  Event Time =Accuracy
• 18.
  Failure Handling
• 19.
  Downtime Handling
• 20.
  Data Reprocessing
• 21.
  We’re Hiring! http://data-artisans.com/careers
• 22.
  Flink Forward 2016,Berlin Submission deadline: June 30, 2016 Early bird deadline: July 15, 2016 www.flink-forward.org
• 23.
  Questions?
• 24.
  Thanks!