Till Rohrmann, profile picture
Uploaded byTill Rohrmann
PDF, PPTX2,720 views

Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015

Apache Flink is an open source platform for distributed stream and batch data processing. It provides APIs called DataStream for unbounded streaming data and DataSet for bounded batch data. Flink runs streaming topologies that allow for windowing, aggregation and other stream processing functions. It supports exactly-once processing semantics through distributed snapshots and checkpoints. The system is optimized for low latency and high throughput streaming applications.

Technology◦
Related topics:
Apache Flink•

Embed presentation

Download as PDF, PPTX
Streaming Data Flow with Apache Flink Till Rohrmann trohrmann@apache.org @stsffap
Recent History April ‘14 December ‘14 v0.5 v0.6 v0.7 April ‘15 Project Incubation Top Level Project v0.8 v0.9 Currently moving towards 0.10 and 1.0 release.
What is Flink? Deployment
 Local (Single JVM) · Cluster (Standalone, YARN) DataStream API Unbounded Data DataSet API Bounded Data Runtime Distributed Streaming Data Flow Libraries Machine Learning · Graph Processing · SQL-like API
What is Flink? Streaming Topologies Stream Time Window Count Low Latency Long Batch Pipelines Resource Utilization 1.2 1.4 1.5 1.2 0.8 0.9 1.0 0.8 Rating Matrix User Matrix Item Matrix 1.5 1.7 1.2 0.6 1.0 1.1 0.8 0.4 W X Y ZW X Y Z A B C D 4.0 4.5 5.0 3.5 2.0 3.5 4.0 2.0 1.0 = X User Machine Learning Iterative Algorithms Graph Analysis 53 1 2 4 0.5 0.2 0.9 0.3 0.1 0.4 0.7 Mutable State
Stream Processing Real world data is unbounded and is pushed to systems. BatchStreaming
Stream Platform Architecture Server Logs Trxn Logs Sensor Logs Downstream Systems Flink – Analyze and correlate streams – Create derived streams Kafka – Gather and backup streams – Offer streams
Cornerstones of Flink Low Latency for fast results. High Throughput to handle many events per second. Exactly-once guarantees for correct results. Expressive APIs for productivity.
sum DataStream API keyBy sumTime Window Time Window
sum DataStream API keyBy sumTime Window Time Window
sum DataStream API keyBy sumTime Window Time Window
sum DataStream API keyBy sumTime Window Time Window
sum DataStream API keyBy sumTime Window Time Window
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API StreamExecutionEnvironment env = StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
DataStream API public static class SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
DataStream API public static class SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
DataStream API public static class SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
DataStream API public static class SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
DataStream API public static class SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
DataStream API public static class SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
Pipelining DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, …); // DataStream WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // split stream by word .sum(1); // sum per word as they arrive Source Map Reduce
Pipelining S1 M1 R1 S2 M2 R2 Source Map Reduce Complete pipeline online concurrently.
Pipelining S1 M1 R1 S2 M2 R2 Chained tasks Complete pipeline online concurrently. Source Map Reduce
Pipelining S1 M1 R1 S2 M2 R2 Chained tasks Complete pipeline online concurrently. Source Map Reduce S1 · M1
Pipelining S1 S2 M2 M1 R1 Complete pipeline online concurrently. Chained tasks Pipelined Shuffle Source Map Reduce S1 · M1 R2
Pipelining Complete pipeline online concurrently. Worker Worker
Pipelining Complete pipeline online concurrently. Worker Worker
Pipelining Complete pipeline online concurrently. Worker Worker
Pipelining Complete pipeline online concurrently. Worker Worker
Pipelining Complete pipeline online concurrently. Worker Worker
Streaming Fault Tolerance At Most Once • No guarantees at all At Least Once • Ensure that all operators see all events. Exactly Once • Ensure that all operators see all events. • Do not perform duplicates updates to operator state. Flink gives you all guarantees.
Distributed Snapshots Barriers flow through the topology in line with data. Flink guarantees exactly once processing. Part of snapshot
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: State 1: Source 2: State 2: Source 3: Sink 1: Source 4: Sink 2: Offset: 6791 Offset: 7252 Offset: 5589 Offset: 6843
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: State 1: Source 2: State 2: Source 3: Sink 1: Source 4: Sink 2: Offset: 6791 Offset: 7252 Offset: 5589 Offset: 6843 Start Checkpoint Message
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: Source 2: 7252 State 2: Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: Emit Barriers Acknowledge with Position
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: Source 2: 7252 State 2: Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: Received barrier at each input
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: Source 2: 7252 State 2: Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: s1 Write snapshot of its state Received barrier at each input
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: PTR1 Source 2: 7252 State 2: PTR2 Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: s1 Acknowledge with pointer to state s2
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: PTR1 Source 2: 7252 State 2: PTR2 Source 3: 5589 Sink 1: ACK Source 4: 6843 Sink 2: ACK s1 s2 Acknowledge Checkpoint Received barrier at each input
Distributed Snapshots Flink guarantees exactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: PTR1 Source 2: 7252 State 2: PTR2 Source 3: 5589 Sink 1: ACK Source 4: 6843 Sink 2: ACK s1 s2
Operator State Stateless Operators ds.filter(_ != 0) System state ds.keyBy(0).window(TumblingTimeWindows.of(5, TimeUnit.SECONDS)) User defined state public class CounterSum implements RichReduceFunction<Long> { private OperatorState<Long> counter; @Override public Long reduce(Long v1, Long v2) throws Exception { counter.update(counter.value() + 1); return v1 + v2; } @Override public void open(Configuration config) { counter = getRuntimeContext().getOperatorState(“counter”, 0L, false); } }
Batch on Streaming DataStream API Unbounded Data DataSet API Bounded Data Runtime Distributed Streaming Data Flow Libraries Machine Learning · Graph Processing · SQL-like API
Batch on Streaming Run a bounded stream (data set) on
 a stream processor. Bounded data set Unbounded data stream
Batch on Streaming Stream Windows Pipelined Data Exchange Global View Pipelined or Blocking Data Exchange Infinite Streams Finite Streams Run a bounded stream (data set) on
 a stream processor.
Batch Pipelines Data exchange
 is mostly streamed Some operators block (e.g. sort, hash table)
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
DataSet API ExecutionEnvironment env = ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
Batch-specific optimizations Cost-based optimizer • Program adapts to changing data size Managed memory • On- and off-heap memory • Internal operators (e.g. join or sort) with out-of-core support • Serialization stack for user-types
Demo Time
Getting Started Project Page: http://flink.apache.org
Getting Started Project Page: http://flink.apache.org Quickstarts: Java & Scala API
Getting Started Project Page: http://flink.apache.org Docs: Programming Guides
Getting Started Project Page: http://flink.apache.org Get Involved: Mailing Lists, Stack Overflow, IRC, …
Blogs http://flink.apache.org/blog http://data-artisans.com/blog Twitter @ApacheFlink Mailing lists (news|user|dev)@flink.apache.org Apache Flink
Thank You!

More Related Content

PPTX
Apache Flink: API, runtime, and project roadmap
byKostas Tzoumas
 
PPTX
Apache Flink Training: DataStream API Part 1 Basic
byFlink Forward
 
PDF
Apache Flink internals
byKostas Tzoumas
 
PDF
Apache Flink Internals: Stream & Batch Processing in One System – Apache Flin...
byucelebi
 
PPTX
Flink Batch Processing and Iterations
bySameer Wadkar
 
PPTX
Flink 0.10 @ Bay Area Meetup (October 2015)
byStephan Ewen
 
PPTX
Apache Flink @ NYC Flink Meetup
byStephan Ewen
 
PDF
Flink Streaming Berlin Meetup
byMárton Balassi
 
Apache Flink: API, runtime, and project roadmap
byKostas Tzoumas
 
Apache Flink Training: DataStream API Part 1 Basic
byFlink Forward
 
Apache Flink internals
byKostas Tzoumas
 
Apache Flink Internals: Stream & Batch Processing in One System – Apache Flin...
byucelebi
 
Flink Batch Processing and Iterations
bySameer Wadkar
 
Flink 0.10 @ Bay Area Meetup (October 2015)
byStephan Ewen
 
Apache Flink @ NYC Flink Meetup
byStephan Ewen
 
Flink Streaming Berlin Meetup
byMárton Balassi
 

What's hot

PDF
Unified Stream and Batch Processing with Apache Flink
byDataWorks Summit/Hadoop Summit
 
PDF
Flink Forward SF 2017: Stefan Richter - Improvements for large state and reco...
byFlink Forward
 
PPTX
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
byFlink Forward
 
PDF
Flink forward SF 2017: Ufuk Celebi - The Stream Processor as a Database: Buil...
byFlink Forward
 
PDF
Machine Learning with Apache Flink at Stockholm Machine Learning Group
byTill Rohrmann
 
PDF
Flink Gelly - Karlsruhe - June 2015
byAndra Lungu
 
PDF
Francesco Versaci - Flink in genomics - efficient and scalable processing of ...
byFlink Forward
 
PDF
Apache Flink Stream Processing
bySuneel Marthi
 
PPTX
Flink internals web
byKostas Tzoumas
 
PDF
Ufuc Celebi – Stream & Batch Processing in one System
byFlink Forward
 
PPTX
Apache Flink@ Strata & Hadoop World London
byStephan Ewen
 
PDF
Apache Flink Deep Dive
byVasia Kalavri
 
PDF
Vasia Kalavri – Training: Gelly School
byFlink Forward
 
PPTX
First Flink Bay Area meetup
byKostas Tzoumas
 
PPTX
Flink Streaming @BudapestData
byGyula Fóra
 
PDF
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
byVasia Kalavri
 
PPTX
Continuous Processing with Apache Flink - Strata London 2016
byStephan Ewen
 
PDF
Flink Forward SF 2017: Stephan Ewen - Experiences running Flink at Very Large...
byFlink Forward
 
PPTX
Debunking Common Myths in Stream Processing
byKostas Tzoumas
 
PPTX
Michael Häusler – Everyday flink
byFlink Forward
 
Unified Stream and Batch Processing with Apache Flink
byDataWorks Summit/Hadoop Summit
 
Flink Forward SF 2017: Stefan Richter - Improvements for large state and reco...
byFlink Forward
 
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
byFlink Forward
 
Flink forward SF 2017: Ufuk Celebi - The Stream Processor as a Database: Buil...
byFlink Forward
 
Machine Learning with Apache Flink at Stockholm Machine Learning Group
byTill Rohrmann
 
Flink Gelly - Karlsruhe - June 2015
byAndra Lungu
 
Francesco Versaci - Flink in genomics - efficient and scalable processing of ...
byFlink Forward
 
Apache Flink Stream Processing
bySuneel Marthi
 
Flink internals web
byKostas Tzoumas
 
Ufuc Celebi – Stream & Batch Processing in one System
byFlink Forward
 
Apache Flink@ Strata & Hadoop World London
byStephan Ewen
 
Apache Flink Deep Dive
byVasia Kalavri
 
Vasia Kalavri – Training: Gelly School
byFlink Forward
 
First Flink Bay Area meetup
byKostas Tzoumas
 
Flink Streaming @BudapestData
byGyula Fóra
 
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
byVasia Kalavri
 
Continuous Processing with Apache Flink - Strata London 2016
byStephan Ewen
 
Flink Forward SF 2017: Stephan Ewen - Experiences running Flink at Very Large...
byFlink Forward
 
Debunking Common Myths in Stream Processing
byKostas Tzoumas
 
Michael Häusler – Everyday flink
byFlink Forward
 

Viewers also liked

PDF
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
PPTX
Apache Flink Training: System Overview
byFlink Forward
 
PPTX
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
byFlink Forward
 
PPTX
Fabian Hueske – Cascading on Flink
byFlink Forward
 
PDF
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
byFlink Forward
 
PDF
Christian Kreuzfeld – Static vs Dynamic Stream Processing
byFlink Forward
 
PDF
Flink Apachecon Presentation
byGyula Fóra
 
PPTX
Apache Flink - Hadoop MapReduce Compatibility
byFabian Hueske
 
PDF
Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose
byFlink Forward
 
PDF
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
byFlink Forward
 
PDF
Suneel Marthi – BigPetStore Flink: A Comprehensive Blueprint for Apache Flink
byFlink Forward
 
PDF
Jim Dowling – Interactive Flink analytics with HopsWorks and Zeppelin
byFlink Forward
 
PDF
K. Tzoumas & S. Ewen – Flink Forward Keynote
byFlink Forward
 
PPTX
Chris Hillman – Beyond Mapreduce Scientific Data Processing in Real-time
byFlink Forward
 
PDF
Matthias J. Sax – A Tale of Squirrels and Storms
byFlink Forward
 
PPTX
Aljoscha Krettek – Notions of Time
byFlink Forward
 
PDF
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
byFlink Forward
 
PPTX
Assaf Araki – Real Time Analytics at Scale
byFlink Forward
 
PPTX
Kamal Hakimzadeh – Reproducible Distributed Experiments
byFlink Forward
 
PDF
Fabian Hueske – Juggling with Bits and Bytes
byFlink Forward
 
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
Apache Flink Training: System Overview
byFlink Forward
 
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
byFlink Forward
 
Fabian Hueske – Cascading on Flink
byFlink Forward
 
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
byFlink Forward
 
Christian Kreuzfeld – Static vs Dynamic Stream Processing
byFlink Forward
 
Flink Apachecon Presentation
byGyula Fóra
 
Apache Flink - Hadoop MapReduce Compatibility
byFabian Hueske
 
Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose
byFlink Forward
 
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
byFlink Forward
 
Suneel Marthi – BigPetStore Flink: A Comprehensive Blueprint for Apache Flink
byFlink Forward
 
Jim Dowling – Interactive Flink analytics with HopsWorks and Zeppelin
byFlink Forward
 
K. Tzoumas & S. Ewen – Flink Forward Keynote
byFlink Forward
 
Chris Hillman – Beyond Mapreduce Scientific Data Processing in Real-time
byFlink Forward
 
Matthias J. Sax – A Tale of Squirrels and Storms
byFlink Forward
 
Aljoscha Krettek – Notions of Time
byFlink Forward
 
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
byFlink Forward
 
Assaf Araki – Real Time Analytics at Scale
byFlink Forward
 
Kamal Hakimzadeh – Reproducible Distributed Experiments
byFlink Forward
 
Fabian Hueske – Juggling with Bits and Bytes
byFlink Forward
 

Similar to Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015

PPTX
Introduction to Apache Flink
bymxmxm
 
PPTX
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
PDF
Java 8 Stream API (Valdas Zigas)
byKaunas Java User Group
 
PPTX
Flink Streaming Hadoop Summit San Jose
byKostas Tzoumas
 
PPTX
Flexible and Real-Time Stream Processing with Apache Flink
byDataWorks Summit
 
PPTX
Kostas Kloudas - Extending Flink's Streaming APIs
byVerverica
 
PPTX
Stream processing - Apache flink
byRenato Guimaraes
 
PDF
Streaming Dataflow with Apache Flink
byhuguk
 
PDF
Spark Streaming with Cassandra
byJacek Lewandowski
 
PPTX
Apache Flink Overview at SF Spark and Friends
byStephan Ewen
 
PDF
Apache flink: data streaming as a basis for all analytics by Kostas Tzoumas a...
byBig Data Spain
 
PDF
Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...
bySpark Summit
 
PDF
Distributed Stream Processing - Spark Summit East 2017
byPetr Zapletal
 
PDF
Distributed Real-Time Stream Processing: Why and How 2.0
byPetr Zapletal
 
PPTX
Flink Forward SF 2017: Konstantinos Kloudas - Extending Flink’s Streaming APIs
byFlink Forward
 
PDF
RxJava applied [JavaDay Kyiv 2016]
byIgor Lozynskyi
 
PDF
Maximilian Michels – Google Cloud Dataflow on Top of Apache Flink
byFlink Forward
 
PPTX
January 2015 HUG: Apache Flink: Fast and reliable large-scale data processing
byYahoo Developer Network
 
PDF
Strumienie i wiewiórka
byDawid Wysakowicz
 
PPTX
Flink meetup
byChristos Hadjinikolis
 
Introduction to Apache Flink
bymxmxm
 
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
Java 8 Stream API (Valdas Zigas)
byKaunas Java User Group
 
Flink Streaming Hadoop Summit San Jose
byKostas Tzoumas
 
Flexible and Real-Time Stream Processing with Apache Flink
byDataWorks Summit
 
Kostas Kloudas - Extending Flink's Streaming APIs
byVerverica
 
Stream processing - Apache flink
byRenato Guimaraes
 
Streaming Dataflow with Apache Flink
byhuguk
 
Spark Streaming with Cassandra
byJacek Lewandowski
 
Apache Flink Overview at SF Spark and Friends
byStephan Ewen
 
Apache flink: data streaming as a basis for all analytics by Kostas Tzoumas a...
byBig Data Spain
 
Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...
bySpark Summit
 
Distributed Stream Processing - Spark Summit East 2017
byPetr Zapletal
 
Distributed Real-Time Stream Processing: Why and How 2.0
byPetr Zapletal
 
Flink Forward SF 2017: Konstantinos Kloudas - Extending Flink’s Streaming APIs
byFlink Forward
 
RxJava applied [JavaDay Kyiv 2016]
byIgor Lozynskyi
 
Maximilian Michels – Google Cloud Dataflow on Top of Apache Flink
byFlink Forward
 
January 2015 HUG: Apache Flink: Fast and reliable large-scale data processing
byYahoo Developer Network
 
Strumienie i wiewiórka
byDawid Wysakowicz
 
Flink meetup
byChristos Hadjinikolis
 

More from Till Rohrmann

PDF
Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015
byTill Rohrmann
 
PPTX
From Apache Flink® 1.3 to 1.4
byTill Rohrmann
 
PDF
Streaming Analytics & CEP - Two sides of the same coin?
byTill Rohrmann
 
PDF
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
byTill Rohrmann
 
PDF
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
byTill Rohrmann
 
PDF
Apache Flink and More @ MesosCon Asia 2017
byTill Rohrmann
 
PDF
Apache Flink: Streaming Done Right @ FOSDEM 2016
byTill Rohrmann
 
PDF
Apache Flink® Meets Apache Mesos® and DC/OS
byTill Rohrmann
 
PDF
Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015
byTill Rohrmann
 
PDF
Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin
byTill Rohrmann
 
PDF
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
byTill Rohrmann
 
PDF
Elastic Streams at Scale @ Flink Forward 2018 Berlin
byTill Rohrmann
 
PPTX
Apache flink 1.7 and Beyond
byTill Rohrmann
 
PPTX
Redesigning Apache Flink's Distributed Architecture @ Flink Forward 2017
byTill Rohrmann
 
PDF
Gilbert: Declarative Sparse Linear Algebra on Massively Parallel Dataflow Sys...
byTill Rohrmann
 
PDF
Scaling stream data pipelines with Pravega and Apache Flink
byTill Rohrmann
 
PDF
Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin
byTill Rohrmann
 
Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015
byTill Rohrmann
 
From Apache Flink® 1.3 to 1.4
byTill Rohrmann
 
Streaming Analytics & CEP - Two sides of the same coin?
byTill Rohrmann
 
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...
byTill Rohrmann
 
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
byTill Rohrmann
 
Apache Flink and More @ MesosCon Asia 2017
byTill Rohrmann
 
Apache Flink: Streaming Done Right @ FOSDEM 2016
byTill Rohrmann
 
Apache Flink® Meets Apache Mesos® and DC/OS
byTill Rohrmann
 
Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015
byTill Rohrmann
 
Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin
byTill Rohrmann
 
Modern Stream Processing With Apache Flink @ GOTO Berlin 2017
byTill Rohrmann
 
Elastic Streams at Scale @ Flink Forward 2018 Berlin
byTill Rohrmann
 
Apache flink 1.7 and Beyond
byTill Rohrmann
 
Redesigning Apache Flink's Distributed Architecture @ Flink Forward 2017
byTill Rohrmann
 
Gilbert: Declarative Sparse Linear Algebra on Massively Parallel Dataflow Sys...
byTill Rohrmann
 
Scaling stream data pipelines with Pravega and Apache Flink
byTill Rohrmann
 
Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin
byTill Rohrmann
 

Recently uploaded

PPTX
Expanding Your Toolbox: Beginners Guide to Controlling Kubernetes Logs
byEric D. Schabell
 
PDF
Schedule Future Tasks - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Automating ArcGIS Enterprise Compliance for Operational Excellence
bySafe Software
 
PDF
Run Containers in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Improve Command Line Productivity - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
November Patch Tuesday
byIvanti
 
PDF
Manage Logical Volume in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
PPTX
Governance, Deployment & Methodologies for Agentic Automation [2/3]
bysuhanisingh58689
 
PDF
Managing SELinux Security - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
ENTSO-E's Response to the European Commission Call for Evidence on the Strate...
byReynir Orn Bachmann Gudmundsson
 
PPTX
Cyber Security Quiz 1st Year CSE CS Students
bynirmalamca
 
PDF
Reset RHEL Root User Password - RHCSA.pdf
byLinuxCert Guru
 
PDF
Upskill to Agentic Automation - Accelerating Your Job Search using AI
byDianaGray10
 
PPTX
Bulwark Pokemon League Top 8 graphic archive
byGc Howard
 
PDF
Digital Garage Certification - Google - Viktor Huszag - Online Marketing Fund...
byViktor Huszag
 
PDF
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
PPTX
How Design Systems and AI Agents Accelerate Product Delivery Sixt
byBoyan Dimitrov
 
PPTX
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
PPTX
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
PDF
Case Study: How LKQ Corporation Improved Training Efficiency & Delivery
byRustici Software
 
Expanding Your Toolbox: Beginners Guide to Controlling Kubernetes Logs
byEric D. Schabell
 
Schedule Future Tasks - RHCSA (RH134).pdf
byLinuxCert Guru
 
Automating ArcGIS Enterprise Compliance for Operational Excellence
bySafe Software
 
Run Containers in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
Improve Command Line Productivity - RHCSA (RH134).pdf
byLinuxCert Guru
 
November Patch Tuesday
byIvanti
 
Manage Logical Volume in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
Governance, Deployment & Methodologies for Agentic Automation [2/3]
bysuhanisingh58689
 
Managing SELinux Security - RHCSA (RH134).pdf
byLinuxCert Guru
 
ENTSO-E's Response to the European Commission Call for Evidence on the Strate...
byReynir Orn Bachmann Gudmundsson
 
Cyber Security Quiz 1st Year CSE CS Students
bynirmalamca
 
Reset RHEL Root User Password - RHCSA.pdf
byLinuxCert Guru
 
Upskill to Agentic Automation - Accelerating Your Job Search using AI
byDianaGray10
 
Bulwark Pokemon League Top 8 graphic archive
byGc Howard
 
Digital Garage Certification - Google - Viktor Huszag - Online Marketing Fund...
byViktor Huszag
 
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
How Design Systems and AI Agents Accelerate Product Delivery Sixt
byBoyan Dimitrov
 
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
Case Study: How LKQ Corporation Improved Training Efficiency & Delivery
byRustici Software
 

Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015

  • 1.
    Streaming Data Flow withApache Flink Till Rohrmann [email protected] @stsffap
  • 2.
    Recent History April ‘14December ‘14 v0.5 v0.6 v0.7 April ‘15 Project Incubation Top Level Project v0.8 v0.9 Currently moving towards 0.10 and 1.0 release.
  • 3.
    What is Flink? Deployment
 Local(Single JVM) · Cluster (Standalone, YARN) DataStream API Unbounded Data DataSet API Bounded Data Runtime Distributed Streaming Data Flow Libraries Machine Learning · Graph Processing · SQL-like API
  • 4.
    What is Flink? Streaming Topologies Stream Time Window Count LowLatency Long Batch Pipelines Resource Utilization 1.2 1.4 1.5 1.2 0.8 0.9 1.0 0.8 Rating Matrix User Matrix Item Matrix 1.5 1.7 1.2 0.6 1.0 1.1 0.8 0.4 W X Y ZW X Y Z A B C D 4.0 4.5 5.0 3.5 2.0 3.5 4.0 2.0 1.0 = X User Machine Learning Iterative Algorithms Graph Analysis 53 1 2 4 0.5 0.2 0.9 0.3 0.1 0.4 0.7 Mutable State
  • 5.
    Stream Processing Real worlddata is unbounded and is pushed to systems. BatchStreaming
  • 6.
    Stream Platform Architecture Server Logs Trxn Logs Sensor Logs Downstream Systems Flink –Analyze and correlate streams – Create derived streams Kafka – Gather and backup streams – Offer streams
  • 7.
    Cornerstones of Flink LowLatency for fast results. High Throughput to handle many events per second. Exactly-once guarantees for correct results. Expressive APIs for productivity.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 14.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 15.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 16.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 17.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 18.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 19.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 20.
    DataStream API StreamExecutionEnvironment env= StreamExecutionEnvironment
 .getExecutionEnvironment() DataStream<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataStream Windowed WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // [word, [1, 1, …]] for 10 seconds .timeWindow(Time.of(10, TimeUnit.SECONDS)) .sum(1); // sum per word per 10 second window counts.print(); env.execute();
  • 21.
    DataStream API public staticclass SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
  • 22.
    DataStream API public staticclass SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
  • 23.
    DataStream API public staticclass SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
  • 24.
    DataStream API public staticclass SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
  • 25.
    DataStream API public staticclass SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
  • 26.
    DataStream API public staticclass SplitByWhitespace
 implements FlatMapFunction<String, Tuple2<String, Integer>> {
 
 @Override
 public void flatMap ( String value, Collector<Tuple2<String, Integer>> out) { 
 String[] tokens = value.toLowerCase().split("W+");
 
 for (String token : tokens) {
 if (token.length() > 0) {
 out.collect(new Tuple2<>(token, 1));
 }
 }
 }
 }
  • 27.
    Pipelining DataStream<String> data =env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, …); // DataStream WordCount DataStream<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .keyBy(0) // split stream by word .sum(1); // sum per word as they arrive Source Map Reduce
  • 28.
    Pipelining S1 M1 R1 S2M2 R2 Source Map Reduce Complete pipeline online concurrently.
  • 29.
    Pipelining S1 M1 R1 S2M2 R2 Chained tasks Complete pipeline online concurrently. Source Map Reduce
  • 30.
    Pipelining S1 M1 R1 S2M2 R2 Chained tasks Complete pipeline online concurrently. Source Map Reduce S1 · M1
  • 31.
    Pipelining S1 S2 M2 M1 R1 Completepipeline online concurrently. Chained tasks Pipelined Shuffle Source Map Reduce S1 · M1 R2
  • 32.
    Pipelining Complete pipeline onlineconcurrently. Worker Worker
  • 33.
    Pipelining Complete pipeline onlineconcurrently. Worker Worker
  • 34.
    Pipelining Complete pipeline onlineconcurrently. Worker Worker
  • 35.
    Pipelining Complete pipeline onlineconcurrently. Worker Worker
  • 36.
    Pipelining Complete pipeline onlineconcurrently. Worker Worker
  • 37.
    Streaming Fault Tolerance AtMost Once • No guarantees at all At Least Once • Ensure that all operators see all events. Exactly Once • Ensure that all operators see all events. • Do not perform duplicates updates to operator state. Flink gives you all guarantees.
  • 38.
    Distributed Snapshots Barriers flowthrough the topology in line with data. Flink guarantees exactly once processing. Part of snapshot
  • 39.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: State 1: Source 2: State 2: Source 3: Sink 1: Source 4: Sink 2: Offset: 6791 Offset: 7252 Offset: 5589 Offset: 6843
  • 40.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: State 1: Source 2: State 2: Source 3: Sink 1: Source 4: Sink 2: Offset: 6791 Offset: 7252 Offset: 5589 Offset: 6843 Start Checkpoint Message
  • 41.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: Source 2: 7252 State 2: Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: Emit Barriers Acknowledge with Position
  • 42.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: Source 2: 7252 State 2: Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: Received barrier at each input
  • 43.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: Source 2: 7252 State 2: Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: s1 Write snapshot of its state Received barrier at each input
  • 44.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: PTR1 Source 2: 7252 State 2: PTR2 Source 3: 5589 Sink 1: Source 4: 6843 Sink 2: s1 Acknowledge with pointer to state s2
  • 45.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: PTR1 Source 2: 7252 State 2: PTR2 Source 3: 5589 Sink 1: ACK Source 4: 6843 Sink 2: ACK s1 s2 Acknowledge Checkpoint Received barrier at each input
  • 46.
    Distributed Snapshots Flink guaranteesexactly once processing. 
 JobManager Master State Backend Checkpoint Data Source 1: 6791 State 1: PTR1 Source 2: 7252 State 2: PTR2 Source 3: 5589 Sink 1: ACK Source 4: 6843 Sink 2: ACK s1 s2
  • 47.
    Operator State Stateless Operators ds.filter(_!= 0) System state ds.keyBy(0).window(TumblingTimeWindows.of(5, TimeUnit.SECONDS)) User defined state public class CounterSum implements RichReduceFunction<Long> { private OperatorState<Long> counter; @Override public Long reduce(Long v1, Long v2) throws Exception { counter.update(counter.value() + 1); return v1 + v2; } @Override public void open(Configuration config) { counter = getRuntimeContext().getOperatorState(“counter”, 0L, false); } }
  • 48.
    Batch on Streaming DataStreamAPI Unbounded Data DataSet API Bounded Data Runtime Distributed Streaming Data Flow Libraries Machine Learning · Graph Processing · SQL-like API
  • 49.
    Batch on Streaming Runa bounded stream (data set) on
 a stream processor. Bounded data set Unbounded data stream
  • 50.
    Batch on Streaming StreamWindows Pipelined Data Exchange Global View Pipelined or Blocking Data Exchange Infinite Streams Finite Streams Run a bounded stream (data set) on
 a stream processor.
  • 51.
    Batch Pipelines Data exchange
 ismostly streamed Some operators block (e.g. sort, hash table)
  • 52.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 53.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 54.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 55.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 56.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 57.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 58.
    DataSet API ExecutionEnvironment env= ExecutionEnvironment
 .getExecutionEnvironment() DataSet<String> data = env.fromElements( "O Romeo, Romeo! wherefore art thou Romeo?”, ...); // DataSet WordCount DataSet<Tuple2<String, Integer>> counts = data .flatMap(new SplitByWhitespace()) // (word, 1) .groupBy(0) // [word, [1, 1, …]] .sum(1); // sum per word for all occurrences counts.print();
  • 59.
    Batch-specific optimizations Cost-based optimizer •Program adapts to changing data size Managed memory • On- and off-heap memory • Internal operators (e.g. join or sort) with out-of-core support • Serialization stack for user-types
  • 60.
  • 61.
    Getting Started Project Page:http://flink.apache.org
  • 62.
    Getting Started Project Page:http://flink.apache.org Quickstarts: Java & Scala API
  • 63.
    Getting Started Project Page:http://flink.apache.org Docs: Programming Guides
  • 64.
    Getting Started Project Page:http://flink.apache.org Get Involved: Mailing Lists, Stack Overflow, IRC, …
  • 65.
  • 66.