Ververica , profile picture
Uploaded byVerverica
646 views

Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

The document discusses a system for real-time stream processing using Flink to join data streams of scientific accounts and publications, highlighting challenges like fault tolerance and accurate joining of data from distributed sources. It outlines a prototype implementation that facilitates the integration of user accounts with their respective publications while addressing issues related to change data capture and data synchronization. The document emphasizes the importance of maintaining up-to-date and accurate join results in a scalable infrastructure supporting millions of publications.

Embed presentation

Downloaded 21 times
Joining Infinity — Windowless Stream Processing with Flink Sanjar Akhmedov, Software Engineer, ResearchGate
It started when two researchers discovered first- hand that collaborating with a friend or colleague on the other side of the world was no easy task. There are many variations of passages of Lorem Ipsum ResearchGate is a social network for scientists.
Connect the world of science. Make research open to all.
Structured system There are many variations of passages of Lorem Ipsum We have, and are continuing to change how scientific knowledge is shared and discovered.
11,000,000+ Members 110,000,000+ Publications 1,300,000,000+ Citations
Feature: Research Timeline
Feature: Research Timeline
Diverse data sources Proxy Frontend Services memcache MongoDB Solr PostgreSQL Infinispan HBaseMongoDB Solr
Big data pipeline Change data capture Import Hadoop cluster Export
Data Model Account Publication Claim 1 * Author Authorship 1*
Hypothetical SQL Publication Authorship 1* CREATE TABLE publications ( id SERIAL PRIMARY KEY, author_ids INTEGER[] ); Account Claim 1 * Author CREATE TABLE accounts ( id SERIAL PRIMARY KEY, claimed_author_ids INTEGER[] ); CREATE MATERIALIZED VIEW account_publications REFRESH FAST ON COMMIT AS SELECT accounts.id AS account_id, publications.id AS publication_id FROM accounts JOIN publications ON ANY (accounts.claimed_author_ids) = ANY (publications.author_ids);
• Data sources are distributed across different DBs • Dataset doesn’t fit in memory on a single machine • Join process must be fault tolerant • Deploy changes fast • Up-to-date join result in near real-time • Join result must be accurate Challenges
Change data capture (CDC) User Microservice DB Request Write Cache Sync Solr/ES Sync HBase/HDFS Sync
Change data capture (CDC) User Microservice DB Request Write Log K2 1 K1 4 Extract
Change data capture (CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø Extract
Change data capture (CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract
Change data capture (CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync
Change data capture (CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync HBase/HDFSSolr/ES
Join two CDC streams into one NoSQL1 SQL Kafka Kafka Flink Streaming Join Kafka NoSQL2
Flink job topology Accounts Stream Join(CoFlatMap) Account Publications Publications Stream … Author 2 Author 1 Author N
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
DataStream<Account> accounts = kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
Example dataflow Account Publications Accounts Alice 2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
Example dataflow Account Publications Accounts Alice 2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
Example dataflow Account Publications Accounts Alice 2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N … (Bob, 1)
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N … (Bob, 1)
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Example dataflow Account Publications Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Example dataflow Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
• ✔ Data sources are distributed across different DBs • ✔ Dataset doesn’t fit in memory on a single machine • ✔ Join process must be fault tolerant • ✔ Deploy changes fast • ✔ Up-to-date join result in near real-time • ? Join result must be accurate Challenges
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … ?
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need previous value
Paper1 gets deleted Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets deleted Account Publications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need K1 here, e.g. K1 = 𝒇(Bob, Paper1)
Paper1 gets updated Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets updated Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
Paper1 gets updated Account Publications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Paper1 gets updated Account Publications K1 (Bob, Paper1) (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N … (Alice, Paper1)
Paper1 gets updated Account Publications K1 (Bob, Paper1) ?? (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N … 2. (Alice, 1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K2 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
Alice claims Paper1 via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K3 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1) Pick correct natural IDs e.g. K3 = 𝒇(Alice, Author1, Paper1)
• Keep previous element state to update previous join result • Stream elements are not domain entities but commands such as delete or upsert • Joined stream must have natural IDs to propagate deletes and updates How to solve deletes and updates
Generic join graph Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM …
Generic join graph Operate on commands Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM …
Memory requirements Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM … Full copy of Accounts stream Full copy of Publications stream Full copy of Accounts stream on left side Full copy of Publications stream on right side
Network load Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM … Reshuffle Reshuffle Network Network
• In addition to handling Kafka traffic we need to reshuffle all data twice over the network • We need to keep two full copies of each joined stream in memory Resource considerations
Questions We are hiring - www.researchgate.net/careers
Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

More Related Content

PPTX
Kostas Kloudas - Extending Flink's Streaming APIs
byVerverica
 
PDF
Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015
byTill Rohrmann
 
PDF
Flink forward SF 2017: Ufuk Celebi - The Stream Processor as a Database: Buil...
byFlink Forward
 
PDF
Flink Forward SF 2017: Jamie Grier - Apache Flink - The latest and greatest
byFlink Forward
 
PDF
Streaming Analytics & CEP - Two sides of the same coin?
byTill Rohrmann
 
PPTX
Flink Forward SF 2017: Konstantinos Kloudas - Extending Flink’s Streaming APIs
byFlink Forward
 
PDF
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache Flink
byVerverica
 
PPTX
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 
Kostas Kloudas - Extending Flink's Streaming APIs
byVerverica
 
Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015
byTill Rohrmann
 
Flink forward SF 2017: Ufuk Celebi - The Stream Processor as a Database: Buil...
byFlink Forward
 
Flink Forward SF 2017: Jamie Grier - Apache Flink - The latest and greatest
byFlink Forward
 
Streaming Analytics & CEP - Two sides of the same coin?
byTill Rohrmann
 
Flink Forward SF 2017: Konstantinos Kloudas - Extending Flink’s Streaming APIs
byFlink Forward
 
Tzu-Li (Gordon) Tai - Stateful Stream Processing with Apache Flink
byVerverica
 
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 

What's hot

PPTX
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
byFlink Forward
 
PPTX
Flink Forward SF 2017: Stephan Ewen - Convergence of real-time analytics and ...
byFlink Forward
 
PPTX
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
byFlink Forward
 
PPTX
Flink Forward Berlin 2017: Fabian Hueske - Using Stream and Batch Processing ...
byFlink Forward
 
PPTX
Apache Flink at Strata San Jose 2016
byKostas Tzoumas
 
PDF
Flink Forward SF 2017: Stefan Richter - Improvements for large state and reco...
byFlink Forward
 
PPTX
Kostas Tzoumas - Apache Flink®: State of the Union and What's Next
byVerverica
 
PDF
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
byTimo Walther
 
PPTX
Stephan Ewen - Experiences running Flink at Very Large Scale
byVerverica
 
PDF
Apache Flink Internals: Stream & Batch Processing in One System – Apache Flin...
byucelebi
 
PPTX
Flink Forward Berlin 2017: Patrick Gunia - Migration of a realtime stats prod...
byFlink Forward
 
PPTX
Apache Flink @ NYC Flink Meetup
byStephan Ewen
 
PPTX
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
byStephan Ewen
 
PDF
Flink Streaming Berlin Meetup
byMárton Balassi
 
PPTX
Apache Flink Berlin Meetup May 2016
byStephan Ewen
 
PDF
A look at Flink 1.2
byStefan Richter
 
PDF
Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
byFlink Forward
 
PDF
Flink Forward SF 2017: Stephan Ewen - Experiences running Flink at Very Large...
byFlink Forward
 
PPTX
Continuous Processing with Apache Flink - Strata London 2016
byStephan Ewen
 
PPTX
Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Fli...
byFlink Forward
 
Flink Forward SF 2017: Timo Walther - Table & SQL API – unified APIs for bat...
byFlink Forward
 
Flink Forward SF 2017: Stephan Ewen - Convergence of real-time analytics and ...
byFlink Forward
 
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
byFlink Forward
 
Flink Forward Berlin 2017: Fabian Hueske - Using Stream and Batch Processing ...
byFlink Forward
 
Apache Flink at Strata San Jose 2016
byKostas Tzoumas
 
Flink Forward SF 2017: Stefan Richter - Improvements for large state and reco...
byFlink Forward
 
Kostas Tzoumas - Apache Flink®: State of the Union and What's Next
byVerverica
 
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
byTimo Walther
 
Stephan Ewen - Experiences running Flink at Very Large Scale
byVerverica
 
Apache Flink Internals: Stream & Batch Processing in One System – Apache Flin...
byucelebi
 
Flink Forward Berlin 2017: Patrick Gunia - Migration of a realtime stats prod...
byFlink Forward
 
Apache Flink @ NYC Flink Meetup
byStephan Ewen
 
The Stream Processor as the Database - Apache Flink @ Berlin buzzwords
byStephan Ewen
 
Flink Streaming Berlin Meetup
byMárton Balassi
 
Apache Flink Berlin Meetup May 2016
byStephan Ewen
 
A look at Flink 1.2
byStefan Richter
 
Flink Forward SF 2017: Kenneth Knowles - Back to Sessions overview
byFlink Forward
 
Flink Forward SF 2017: Stephan Ewen - Experiences running Flink at Very Large...
byFlink Forward
 
Continuous Processing with Apache Flink - Strata London 2016
byStephan Ewen
 
Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Fli...
byFlink Forward
 

Viewers also liked

PPTX
Taking a look under the hood of Apache Flink's relational APIs.
byFabian Hueske
 
PPTX
ApacheCon: Apache Flink - Fast and Reliable Large-Scale Data Processing
byFabian Hueske
 
PPTX
Stephan Ewen - Stream Processing as a Foundational Paradigm and Apache Flink'...
byVerverica
 
PDF
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
byDan Halperin
 
PPTX
Kostas Tzoumas - Stream Processing with Apache Flink®
byVerverica
 
PDF
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
byVerverica
 
PPTX
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
bySlim Baltagi
 
PDF
Alexander Kolb – Flink. Yet another Streaming Framework?
byFlink Forward
 
PPTX
Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry
byconfluent
 
PPTX
Robert Metzger - Apache Flink Community Updates November 2016 @ Berlin Meetup
byVerverica
 
PPTX
Stream Analytics with SQL on Apache Flink
byFabian Hueske
 
PPTX
Juggling with Bits and Bytes - How Apache Flink operates on binary data
byFabian Hueske
 
PDF
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
byTill Rohrmann
 
PDF
K. Tzoumas & S. Ewen – Flink Forward Keynote
byFlink Forward
 
PDF
Real-time analytics as a service at King
byGyula Fóra
 
PDF
Stefan Richter - A look at Flink 1.2 and beyond @ Berlin Meetup
byVerverica
 
PPTX
Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...
byVerverica
 
PDF
Dive into Spark Streaming
byGerard Maas
 
PPTX
Apache Flink Community Updates November 2016 @ Berlin Meetup
byRobert Metzger
 
PDF
Tran Nam-Luc – Stale Synchronous Parallel Iterations on Flink
byFlink Forward
 
Taking a look under the hood of Apache Flink's relational APIs.
byFabian Hueske
 
ApacheCon: Apache Flink - Fast and Reliable Large-Scale Data Processing
byFabian Hueske
 
Stephan Ewen - Stream Processing as a Foundational Paradigm and Apache Flink'...
byVerverica
 
Introduction to Apache Beam & No Shard Left Behind: APIs for Massive Parallel...
byDan Halperin
 
Kostas Tzoumas - Stream Processing with Apache Flink®
byVerverica
 
Timo Walther - Table & SQL API - unified APIs for batch and stream processing
byVerverica
 
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
bySlim Baltagi
 
Alexander Kolb – Flink. Yet another Streaming Framework?
byFlink Forward
 
Fundamentals of Stream Processing with Apache Beam, Tyler Akidau, Frances Perry
byconfluent
 
Robert Metzger - Apache Flink Community Updates November 2016 @ Berlin Meetup
byVerverica
 
Stream Analytics with SQL on Apache Flink
byFabian Hueske
 
Juggling with Bits and Bytes - How Apache Flink operates on binary data
byFabian Hueske
 
Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...
byTill Rohrmann
 
K. Tzoumas & S. Ewen – Flink Forward Keynote
byFlink Forward
 
Real-time analytics as a service at King
byGyula Fóra
 
Stefan Richter - A look at Flink 1.2 and beyond @ Berlin Meetup
byVerverica
 
Aljoscha Krettek - Apache Flink for IoT: How Event-Time Processing Enables Ea...
byVerverica
 
Dive into Spark Streaming
byGerard Maas
 
Apache Flink Community Updates November 2016 @ Berlin Meetup
byRobert Metzger
 
Tran Nam-Luc – Stale Synchronous Parallel Iterations on Flink
byFlink Forward
 

Similar to Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

PPTX
Stream processing - Apache flink
byRenato Guimaraes
 
PDF
Deeply Declarative Data Pipelines
byHostedbyConfluent
 
PPTX
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...
byFlink Forward
 
PPTX
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
PPTX
Advanced
bymxmxm
 
PDF
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
PDF
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
byFlink Forward
 
PDF
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
byHostedbyConfluent
 
PDF
CDC Stream Processing with Apache Flink
byTimo Walther
 
PPTX
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
PDF
Changelog Stream Processing with Apache Flink
byFlink Forward
 
PDF
Apache Flink Stream Processing
bySuneel Marthi
 
PDF
Tapping into Scientific Data with Hadoop and Flink
byMichael Häusler
 
PPTX
January 2016 Flink Community Update & Roadmap 2016
byRobert Metzger
 
PPTX
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
PDF
Apache Flink - a Gentle Start
byLiangjun Jiang
 
PDF
Streaming Dataflow with Apache Flink
byhuguk
 
PPTX
Real Time Visibility with Flink
byRafi Aroch
 
PDF
Open west 2015 talk ben coverston
bybcoverston
 
PDF
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
byconfluent
 
Stream processing - Apache flink
byRenato Guimaraes
 
Deeply Declarative Data Pipelines
byHostedbyConfluent
 
Flink Forward SF 2017: David Hardwick, Sean Hester & David Brelloch - Dynami...
byFlink Forward
 
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
Advanced
bymxmxm
 
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
Flink Forward Berlin 2018: Xingcan Cui - "Stream Join in Flink: from Discrete...
byFlink Forward
 
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
byHostedbyConfluent
 
CDC Stream Processing with Apache Flink
byTimo Walther
 
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
Changelog Stream Processing with Apache Flink
byFlink Forward
 
Apache Flink Stream Processing
bySuneel Marthi
 
Tapping into Scientific Data with Hadoop and Flink
byMichael Häusler
 
January 2016 Flink Community Update & Roadmap 2016
byRobert Metzger
 
Flink Forward San Francisco 2019: Build a Table-centric Apache Flink Ecosyste...
byFlink Forward
 
Apache Flink - a Gentle Start
byLiangjun Jiang
 
Streaming Dataflow with Apache Flink
byhuguk
 
Real Time Visibility with Flink
byRafi Aroch
 
Open west 2015 talk ben coverston
bybcoverston
 
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
byconfluent
 

More from Ververica

PPTX
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
byVerverica
 
PPTX
Webinar: Flink SQL in Action - Fabian Hueske
byVerverica
 
PDF
Aljoscha Krettek - Apache Flink® and IoT: How Stateful Event-Time Processing ...
byVerverica
 
PPTX
Kostas Kloudas - Complex Event Processing with Flink: the state of FlinkCEP
byVerverica
 
PDF
Deploying Flink on Kubernetes - David Anderson
byVerverica
 
PDF
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
byVerverica
 
PDF
Webinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin Knauf
byVerverica
 
PDF
Aljoscha Krettek - Portable stateful big data processing in Apache Beam
byVerverica
 
PPTX
Keynote: Stephan Ewen - Stream Processing as a Foundational Paradigm and Apac...
byVerverica
 
PPTX
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 
PDF
2018-01 Seattle Apache Flink Meetup at OfferUp, Opening Remarks and Talk 2
byVerverica
 
PDF
Webinar: How to contribute to Apache Flink - Robert Metzger
byVerverica
 
PDF
Webinar: Detecting row patterns with Flink SQL - Dawid Wysakowicz
byVerverica
 
PDF
2020-05-06 Apache Flink Meetup London: The Easiest Way to Get Operational wit...
byVerverica
 
2018-04 Kafka Summit London: Stephan Ewen - "Apache Flink and Apache Kafka fo...
byVerverica
 
Webinar: Flink SQL in Action - Fabian Hueske
byVerverica
 
Aljoscha Krettek - Apache Flink® and IoT: How Stateful Event-Time Processing ...
byVerverica
 
Kostas Kloudas - Complex Event Processing with Flink: the state of FlinkCEP
byVerverica
 
Deploying Flink on Kubernetes - David Anderson
byVerverica
 
Webinar: Deep Dive on Apache Flink State - Seth Wiesman
byVerverica
 
Webinar: 99 Ways to Enrich Streaming Data with Apache Flink - Konstantin Knauf
byVerverica
 
Aljoscha Krettek - Portable stateful big data processing in Apache Beam
byVerverica
 
Keynote: Stephan Ewen - Stream Processing as a Foundational Paradigm and Apac...
byVerverica
 
Fabian Hueske - Stream Analytics with SQL on Apache Flink
byVerverica
 
2018-01 Seattle Apache Flink Meetup at OfferUp, Opening Remarks and Talk 2
byVerverica
 
Webinar: How to contribute to Apache Flink - Robert Metzger
byVerverica
 
Webinar: Detecting row patterns with Flink SQL - Dawid Wysakowicz
byVerverica
 
2020-05-06 Apache Flink Meetup London: The Easiest Way to Get Operational wit...
byVerverica
 

Recently uploaded

PPTX
ORACLE ADMIN WORKSHOP PRESENTATION OCP.pptx
bykausaratg
 
PDF
Data Visualization That Drives Action.pdf
byseoxbyteanalytics
 
PDF
wow super , fantastic booooo andtha pekuuu acunetics - testhtml5 Report.pdf
byferov57646
 
PDF
analisis mediante tecnicas de inteligencia artificial
byJULYANDREAGOMEZCAMPE
 
PPTX
Hybrid_AI_Dynamic_Pricing_Presentation.pptx
bymnasar3
 
PPT
PROJECT SCHEDIULING ABOUT SOFTWARE DEVELOPMENT ACTIVITIES
bypriyankaR797112
 
PPTX
Energy-Aware Federated Learning Framewor
bymnasar3
 
PPTX
sugar industry deep introducton, flow process, equipments, each parts specifi...
byHamza Ghaffar
 
PPTX
SALONI-ASPIRING-DATA-ANALYST MBA STUDENT.ppt
bySaloniNarwal
 
PDF
Buy Verified Cash App Accounts - BTC Enable Verified 2024.pdf
bycucfuc315
 
PPTX
Laptop Price Prediction for Smart Tech Co (S11666).pptx
byABHINAV GUNTUKU
 
PDF
Garden-Life-Brand-Launch-Case-Study-By-Accelero-Agency.pdf
byacceleroagency
 
PDF
Comprehensive Market Analysis of ESPN with Strategic Recommendations.pdf
byKelishaCyrus
 
DOCX
closing meeting.docx Audit Closing Meeting
bymehmikomal21
 
PDF
How Has Urban Development Affected Living Conditions in Omani Cities?
bynzainb652
 
PPTX
UCSP_Module_1_Presentation culture and society.pptx
byakcsbutuan
 
PDF
TRIPTICO INGLES GRUPAL (1).pdf SOBRE EMBARAZO
byDayanaEspinoza37
 
PPTX
Health Insight Survey: an introduction | Accessible slides
byOffice for National Statistics
 
PPTX
Ritik Jangid Portfolio, a business analyst
byjangidritik851
 
PPT
monocot-vs-dicot.ppt FOR I'M IRRIGATION I LOVE THIS SO BAD BUT THIS ONE WAS
bydaaboulaziz5
 
ORACLE ADMIN WORKSHOP PRESENTATION OCP.pptx
bykausaratg
 
Data Visualization That Drives Action.pdf
byseoxbyteanalytics
 
wow super , fantastic booooo andtha pekuuu acunetics - testhtml5 Report.pdf
byferov57646
 
analisis mediante tecnicas de inteligencia artificial
byJULYANDREAGOMEZCAMPE
 
Hybrid_AI_Dynamic_Pricing_Presentation.pptx
bymnasar3
 
PROJECT SCHEDIULING ABOUT SOFTWARE DEVELOPMENT ACTIVITIES
bypriyankaR797112
 
Energy-Aware Federated Learning Framewor
bymnasar3
 
sugar industry deep introducton, flow process, equipments, each parts specifi...
byHamza Ghaffar
 
SALONI-ASPIRING-DATA-ANALYST MBA STUDENT.ppt
bySaloniNarwal
 
Buy Verified Cash App Accounts - BTC Enable Verified 2024.pdf
bycucfuc315
 
Laptop Price Prediction for Smart Tech Co (S11666).pptx
byABHINAV GUNTUKU
 
Garden-Life-Brand-Launch-Case-Study-By-Accelero-Agency.pdf
byacceleroagency
 
Comprehensive Market Analysis of ESPN with Strategic Recommendations.pdf
byKelishaCyrus
 
closing meeting.docx Audit Closing Meeting
bymehmikomal21
 
How Has Urban Development Affected Living Conditions in Omani Cities?
bynzainb652
 
UCSP_Module_1_Presentation culture and society.pptx
byakcsbutuan
 
TRIPTICO INGLES GRUPAL (1).pdf SOBRE EMBARAZO
byDayanaEspinoza37
 
Health Insight Survey: an introduction | Accessible slides
byOffice for National Statistics
 
Ritik Jangid Portfolio, a business analyst
byjangidritik851
 
monocot-vs-dicot.ppt FOR I'M IRRIGATION I LOVE THIS SO BAD BUT THIS ONE WAS
bydaaboulaziz5
 

Apache Flink Meetup: Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink

  • 1.
    Joining Infinity —Windowless Stream Processing with Flink Sanjar Akhmedov, Software Engineer, ResearchGate
  • 2.
    It started whentwo researchers discovered first- hand that collaborating with a friend or colleague on the other side of the world was no easy task. There are many variations of passages of Lorem Ipsum ResearchGate is a social network for scientists.
  • 3.
    Connect the worldof science. Make research open to all.
  • 4.
    Structured system There aremany variations of passages of Lorem Ipsum We have, and are continuing to change how scientific knowledge is shared and discovered.
  • 5.
  • 6.
  • 7.
  • 8.
    Diverse data sources Proxy Frontend Services memcacheMongoDB Solr PostgreSQL Infinispan HBaseMongoDB Solr
  • 9.
  • 10.
  • 11.
    Hypothetical SQL Publication Authorship 1* CREATE TABLEpublications ( id SERIAL PRIMARY KEY, author_ids INTEGER[] ); Account Claim 1 * Author CREATE TABLE accounts ( id SERIAL PRIMARY KEY, claimed_author_ids INTEGER[] ); CREATE MATERIALIZED VIEW account_publications REFRESH FAST ON COMMIT AS SELECT accounts.id AS account_id, publications.id AS publication_id FROM accounts JOIN publications ON ANY (accounts.claimed_author_ids) = ANY (publications.author_ids);
  • 12.
    • Data sourcesare distributed across different DBs • Dataset doesn’t fit in memory on a single machine • Join process must be fault tolerant • Deploy changes fast • Up-to-date join result in near real-time • Join result must be accurate Challenges
  • 13.
    Change data capture(CDC) User Microservice DB Request Write Cache Sync Solr/ES Sync HBase/HDFS Sync
  • 14.
    Change data capture(CDC) User Microservice DB Request Write Log K2 1 K1 4 Extract
  • 15.
    Change data capture(CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø Extract
  • 16.
    Change data capture(CDC) User Microservice DB Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract
  • 17.
    Change data capture(CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync
  • 18.
    Change data capture(CDC) User Microservice DB Cache Request Write Log K2 1 K1 4 K1 Ø KN 42 … Extract Sync HBase/HDFSSolr/ES
  • 19.
    Join two CDCstreams into one NoSQL1 SQL Kafka Kafka Flink Streaming Join Kafka NoSQL2
  • 20.
  • 21.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 22.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 23.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 24.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 25.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 26.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 27.
    DataStream<Account> accounts =kafkaTopic("accounts"); DataStream<Publication> publications = kafkaTopic("publications"); DataStream<AccountPublication> result = accounts.connect(publications) .keyBy("claimedAuthorId", "publicationAuthorId") .flatMap(new RichCoFlatMapFunction<Account, Publication, AccountPublication>() { transient ValueState<String> authorAccount; transient ValueState<String> authorPublication; public void open(Configuration parameters) throws Exception { authorAccount = getRuntimeContext().getState(new ValueStateDescriptor<>("authorAccount", String.class, null)); authorPublication = getRuntimeContext().getState(new ValueStateDescriptor<>("authorPublication", String.class, null)); } public void flatMap1(Account account, Collector<AccountPublication> out) throws Exception { authorAccount.update(account.id); if (authorPublication.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } public void flatMap2(Publication publication, Collector<AccountPublication> out) throws Exception { authorPublication.update(publication.id); if (authorAccount.value() != null) { out.collect(new AccountPublication(authorAccount.value(), authorPublication.value())); } } }); Prototype implementation
  • 28.
    Example dataflow Account Publications Accounts Alice2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
  • 29.
    Example dataflow Account Publications Accounts Alice2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Author N …
  • 30.
    Example dataflow Account Publications Accounts Alice2 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
  • 31.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N …
  • 32.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Author 2 Alice Author N … (Bob, 1)
  • 33.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N … (Bob, 1)
  • 34.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
  • 35.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Author 2 Alice Author N …
  • 36.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 37.
    Example dataflow Account Publications Accounts Alice2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 38.
    Example dataflow Account Publications K1(Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 39.
    • ✔ Datasources are distributed across different DBs • ✔ Dataset doesn’t fit in memory on a single machine • ✔ Join process must be fault tolerant • ✔ Deploy changes fast • ✔ Up-to-date join result in near real-time • ? Join result must be accurate Challenges
  • 40.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 41.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 42.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … ?
  • 43.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need previous value
  • 44.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 45.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 46.
    Paper1 gets deleted AccountPublications K1 (Bob, Paper1) K1 Ø Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 Ø Accounts Stream Join Account Publications Diff with Previous State Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N … Need K1 here, e.g. K1 = 𝒇(Bob, Paper1)
  • 47.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 48.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Author N …
  • 49.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 50.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N … (Alice, Paper1)
  • 51.
    Paper1 gets updated AccountPublications K1 (Bob, Paper1) ?? (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 2 Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 52.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 53.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 54.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Bob Paper1 Author 2 Alice Paper1 Author N …
  • 55.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
  • 56.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
  • 57.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N …
  • 58.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Ø Paper1 Author 2 Alice Paper1 Author N … 2. (Alice, 1)
  • 59.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1)
  • 60.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
  • 61.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K2 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1)
  • 62.
    Alice claims Paper1via different author Account Publications K1 (Bob, Paper1) K2 (Alice, Paper1) K1 Ø K3 (Alice, Paper1) K2 Ø Accounts Alice 2 Bob 1 Bob Ø Alice 1 Publications Paper1 1 Paper1 (1, 2) Accounts Stream Join Account Publications Publications Stream Author 1 Alice Paper1 Author 2 Ø Paper1 Author N … 2. (Alice, 1) (Alice, Paper1) Pick correct natural IDs e.g. K3 = 𝒇(Alice, Author1, Paper1)
  • 63.
    • Keep previouselement state to update previous join result • Stream elements are not domain entities but commands such as delete or upsert • Joined stream must have natural IDs to propagate deletes and updates How to solve deletes and updates
  • 64.
  • 65.
    Generic join graph Operateon commands Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM …
  • 66.
    Memory requirements Account Publications Accounts Stream Publications Stream Diff Alice Bob … Diff Paper1 PaperN … Join Author1 AuthorM … Full copyof Accounts stream Full copy of Publications stream Full copy of Accounts stream on left side Full copy of Publications stream on right side
  • 67.
  • 68.
    • In additionto handling Kafka traffic we need to reshuffle all data twice over the network • We need to keep two full copies of each joined stream in memory Resource considerations
  • 69.
    Questions We are hiring- www.researchgate.net/careers