Flink Forward, profile picture
Uploaded byFlink Forward
PDF, PPTX9,828 views

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

The document discusses Gradoop, a framework for scalable graph analytics using Apache Flink. It provides an overview of the Gradoop architecture, which includes an extended property graph data model, graph operators implemented as Flink operators, and a DSL for declaring analytical workflows. The architecture supports end-to-end graph analytics, including data integration, operator execution, and result representation.

Technology◦
Related topics:
Apache Flink•

Embed presentation

Download as PDF, PPTX
GRADOOP: Scalable Graph Analytics with Apache Flink Martin Junghanns University of Leipzig
About the speaker and the team  2011 Bachelor of Engineering  Thesis: Partitioning of Dynamic Graphs  2014 Master of Science  Thesis: Graph Database Systems for Business Intelligence  Now: PhD Student, Database Group, University of Leipzig  Distributed Systems  Distributed Graph Data Management  Graph Theory & Algorithms  Professional Experience: sones GraphDB, SAP André, PhD Student Martin, PhD Student Kevin, M.Sc. StudentNiklas, M.Sc. Student
Motivation
𝑮𝑟𝑟𝑟𝑟 = (𝑽𝑒𝑒𝑒𝑒𝑒𝑒𝑒, 𝑬𝑑𝑑𝑑𝑑) “Graphs are everywhere”
𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔, 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔, 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔, 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔, 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔, 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy Trent
𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔, 𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy Trent
𝐺𝐺𝐺𝐺𝐺 = (𝐂𝐂𝐂𝐂𝐂𝐂, 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶) “Graphs are everywhere” Leipzig pop: 544K Dresden pop: 536K Berlin pop: 3.5M Hamburg pop: 1.7M Munich pop: 1.4M Chemnitz pop: 243K Nuremberg pop: 500K Cologne pop: 1M
 World Wide Web  ca. 1 billion websites “Graphs are large”  Facebook  ca. 1.49 billion active users  ca. 340 friends per user
End-to-End Graph Analytics Data Integration Graph Analytics Representation
End-to-End Graph Analytics Data Integration Graph Analytics Representation  Integrate data from one or more sources into a dedicated graph storage with common graph data model
End-to-End Graph Analytics Data Integration Graph Analytics Representation  Integrate data from one or more sources into a dedicated graph storage with common graph data model  Definition of analytical workflows from operator algebra
End-to-End Graph Analytics Data Integration Graph Analytics Representation  Integrate data from one or more sources into a dedicated graph storage with common graph data model  Definition of analytical workflows from operator algebra  Result representation in a meaningful way
Graph Data Management Graph Database Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
Graph Data Management Graph Database Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
Graph Data Management Graph Database Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
Graph Data Management Graph Database Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
What‘s missing? An end-to-end framework and research platform for efficient, distributed and domain independent graph data management and analytics.
What‘s missing? An end-to-end framework and research platform for efficient, distributed and domain independent graph data management and analytics.
Gradoop Architecture & Data Model
High Level Architecture HDFS/YARN Cluster HBase Distributed Graph Store Extended Property Graph Model Flink Operator Implementations Data Integration Flink Operator Execution Workflow Declaration Visual GrALa DSL Representation Data flow Control flow Graph Analytics Representation Workflow Execution
High Level Architecture HBase Distributed Graph Store Extended Property Graph Model Flink Operator Implementations Data Integration Flink Operator Execution Workflow Declaration Visual GrALa DSL Representation Data flow Control flow Graph Analytics Representation Workflow Execution HDFS/YARN Cluster
Extended Property Graph Model
Extended Property Graph Model
Extended Property Graph Model
Graph Operators Operator GrALa notation Binary Combination graph.combine(otherGraph) : Graph Overlap graph.overlap(otherGraph) : Graph Exclusion graph.exclude(otherGraph) : Graph Isomorphism graph.isIsomorphicTo(otherGraph) : Boolean Unary Pattern Matching graph.match(patternGraph,predicate) : Collection Aggregation graph.aggregate(propertyKey,aggregateFunction) : Graph Projection graph.project(vertexFunction,edgeFunction) : Graph Summarization graph.summarize( vertexGroupKeys,vertexAggregateFunction, edgeGroupKeys,edgeAggregateFunction) : Graph
Combination 1: personGraph = db.G[0].combine(db.G[1]).combine(db.G[2])
Combination 1: personGraph = db.G[0].combine(db.G[1]).combine(db.G[2])
Graph Operators Operator GrALa notation Binary Combination graph.combine(otherGraph) : Graph Overlap graph.overlap(otherGraph) : Graph Exclusion graph.exclude(otherGraph) : Graph Isomorphism graph.isIsomorphicTo(otherGraph) : Boolean Unary Pattern Matching graph.match(patternGraph,predicate) : Collection Aggregation graph.aggregate(propertyKey,aggregateFunction) : Graph Projection graph.project(vertexFunction,edgeFunction) : Graph Summarization graph.summarize( vertexGroupKeys,vertexAggregateFunction, edgeGroupKeys,edgeAggregateFunction) : Graph
Summarization 1: personGraph = db.G[0].combine(db.G[1]).combine(db.G[2]) 2: vertexGroupingKeys = {:type, “city”} 3: edgeGroupingKeys = {:type} 4: vertexAggFunc = (Vertex vSum, Set vertices => vSum[“count”] = |vertices|) 5: edgeAggFunc = (Edge eSum, Set edges => eSum[“count”] = |edges|) 6: sumGraph = personGraph.summarize(vertexGroupingKeys, vertexAggFunc, edgeGroupingKeys, edgeAggFunc)
Summarization 1: personGraph = db.G[0].combine(db.G[1]).combine(db.G[2]) 2: vertexGroupingKeys = {:type, “city”} 3: edgeGroupingKeys = {:type} 4: vertexAggFunc = (Vertex vSum, Set vertices => vSum[“count”] = |vertices|) 5: edgeAggFunc = (Edge eSum, Set edges => eSum[“count”] = |edges|) 6: sumGraph = personGraph.summarize(vertexGroupingKeys, vertexAggFunc, edgeGroupingKeys, edgeAggFunc)
Graph Collection Operators Operator GrALa notation Collection Selection collection.select(predicate) : Collection Distinct collection.distinct() : Collection Sort by collection.sortBy(key, [:asc|:desc]) : Collection Top collection.top(limit) : Collection Union collection.union(otherCollection) : Collection Intersection collection.intersect(otherCollection) : Collection Difference collection.difference(otherCollection) : Collection Auxiliary Apply collection.apply(unaryGraphOperator) : Collection Reduce collection.reduce(binaryGraphOperator) : Graph Call [graph|collection].callFor[Graph|Collection]( algorithm,parameters) : [Graph|Collection]
Selection 1: collection = <db.G[0],db.G[1],db.G[2]> 2: predicate = (Graph g => |g.V| > 3) 3: result = collection.select(predicate)
Selection 1: collection = <db.G[0],db.G[1],db.G[2]> 2: predicate = (Graph g => |g.V| > 3) 3: result = collection.select(predicate)
Graph Collection Operators Operator GrALa notation Collection Selection collection.select(predicate) : Collection Distinct collection.distinct() : Collection Sort by collection.sortBy(key, [:asc|:desc]) : Collection Top collection.top(limit) : Collection Union collection.union(otherCollection) : Collection Intersection collection.intersect(otherCollection) : Collection Difference collection.difference(otherCollection) : Collection Auxiliary Apply collection.apply(unaryGraphOperator) : Collection Reduce collection.reduce(binaryGraphOperator) : Graph Call [graph|collection].callFor[Graph|Collection]( algorithm,parameters) : [Graph|Collection]
Extended Property Graph Model in Flink ID Label Properties Graphs ID Label Properties Source Vertex Target Vertex Graphs VertexData EdgeData GraphData ID Label Properties POJO POJO POJO DataSet<Vertex<ID,VertexData>> DataSet<Edge<ID,EdgeData>> DataSet<Subgraph<ID,GraphData>> Gelly 𝒱 ℰ 𝒢 Pojo Representation
Extended Property Graph Model in Flink VertexData EdgeData GraphData POJO POJO POJO DataSet<Vertex<ID,VertexData>> DataSet<Edge<ID,EdgeData>> DataSet<Subgraph<ID,GraphData>> Gelly VertexData EdgeData GraphData Tuple Tuple Tuple DataSet<VertexData> DataSet<EdgeData> DataSet<GraphData> 𝒱 𝒱 ℰ ℰ 𝒢 𝒢 Pojo Representation Tuple Representation ID Label Properties Graphs ID Label Properties Source Vertex Target Vertex Graphs ID Label Properties ID Label Properties Graphs ID Label Properties Source Vertex Target Vertex Graphs ID Label Properties
Summarization in Flink VID City 0 L 1 L 2 D 3 D 4 D 5 B EID S T 0 0 1 1 1 0 2 1 2 3 2 1 4 2 3 5 3 2 6 4 0 7 4 1 8 5 2 9 5 3 L [0,1] D [2,3,4] B [5] VID City Count 0 L 2 2 D 3 5 B 1 VID Rep 0 0 1 0 2 2 3 2 4 2 5 5 ID S T 0 0 1 1 0 0 2 0 2 3 2 1 4 2 3 5 2 2 6 2 0 7 2 1 8 5 2 9 5 3 ID S T 0 0 0 1 0 0 2 0 2 3 2 0 4 2 2 5 2 2 6 2 0 7 2 0 8 5 2 9 5 2 0,0 [0,1] 0,2 [2] 2,0 [3,6,7] 2,2 [4,5] 5,2 [8,9] EID S T Count 0 0 1 2 2 0 2 1 3 2 0 3 4 2 2 2 8 5 2 2 join(VID==S) 𝒱 ℰ’ 𝒱′ ℰ groupBy(City) reduceGroup + filter + map reduceGroup + filter + map groupBy(S,T) join(VID==T)
Use Case: Graph Business Intelligence
Use Case: Graph Business Intelligence  Business intelligence usually based on relational data warehouses  Enterprise data is integrated within dimensional schema  Analysis limited to predefined relationships  No support for relationship-oriented data mining Facts Dim 1 Dim 2 Dim 3
Use Case: Graph Business Intelligence  Business intelligence usually based on relational data warehouses  Enterprise data is integrated within dimensional schema  Analysis limited to predefined relationships  No support for relationship-oriented data mining  Graph-based approach  Integrate data sources within an instance graph by preserving original relationships between data objects (transactional and master data)  Determine subgraphs (business transaction graphs) related to business activities  Analyze subgraphs or entire graphs with aggregation queries, mining relationship patterns, etc. Facts Dim 1 Dim 2 Dim 3
Prerequisites: Data Integration
Business Transaction Graphs CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy basedOn serves serves bills bills bills processedBy
Business Transaction Graphs CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
Business Transaction Graphs CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
Business Transaction Graphs CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
Business Transaction Graphs CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
Business Transaction Graphs CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
BTG 1 (1) BTG Extraction BTG 2 BTG 3 BTG 4 BTG 5 BTG n …
(1) BTG Extraction // generate base collection btgs = iig.callForCollection( :BusinessTransactionGraphs , {} )
(2) Profit Aggregation CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
(2) Profit Aggregation // generate base collection btgs = iig.callForCollection( :BusinessTransactionGraphs , {} ) // define profit aggregate function aggFunc = ( Graph g => g.V.values(“Revenue").sum() - g.V.values(“Expense").sum() )
(2) Profit Aggregation BTG 1 BTG 2 BTG 3 BTG 4 BTG 5 BTG n … ∑ Revenue ∑ Expenses Net Profit 5,000 -3,000 2,000 9,000 -3,000 6,000 2,000 -1,500 500 5,000 -7,000 -2,000 10,000 -15,000 -5,000 … … … 8,000 -4,000 4,000
(2) Profit Aggregation // generate base collection btgs = iig.callForCollection( :BusinessTransactionGraphs , {} ) // define profit aggregate function aggFunc = ( Graph g => g.V.values(“Revenue").sum() - g.V.values(“Expense").sum() ) // apply aggregate function and store result at new property btgs = btgs.apply( Graph g => g.aggregate( “Profit“ , aggFunc ) )
(3) BTG Clustering BTG 1 BTG 2 BTG 3 BTG 4 BTG 5 BTG n … ∑ Revenue ∑ Expenses Net Profit 5,000 -3,000 2,000 9,000 -3,000 6,000 2,000 -1,500 500 5,000 -7,000 -2,000 10,000 -15,000 -5,000 … … … 8,000 -4,000 4,000
(3) BTG Clustering // select profit and loss clusters profitBtgs = btgs.select( Graph g => g[“Profit”] >= 0 ) lossBtgs = btgs.difference(profitBtgs)
(4) Cluster Characteristic Patterns CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
(4) Cluster Characteristic Patterns CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
(4) Cluster Characteristic Patterns BTG 1 BTG 2 BTG 3 BTG 4 BTG 5 BTG n … ∑ Revenue ∑ Expenses Net Profit 5,000 -3,000 2,000 9,000 -3,000 6,000 2,000 -1,500 500 5,000 -7,000 -2,000 10,000 -15,000 -5,000 … … … 8,000 -4,000 4,000 TicketAlice processedBy Bob createdBy PurchaseOrder
(4) Cluster Characteristic Patterns // select profit and loss clusters profitBtgs = btgs.select( Graph g => g[“Profit”] >= 0 ) lossBtgs = btgs.difference(profitBtgs) // apply magic profitFreqPats = profitBtgs.callForCollection( :FrequentSubgraphs , {“Threshold”:0.7} ) lossFreqPats = lossBtgs.callForCollection( :FrequentSubgraphs , {“Threshold”:0.7} ) // determine cluster characteristic patterns trivialPats = profitFreqPats.intersect(lossFreqPats) profitCharPatterns = profitFreqPats.difference(trivialPats) lossCharPatterns = lossFreqPats.difference(trivialPats)
Current State & Future Work
Current State  0.0.1 First Prototype (May 2015)  Hadoop MapReduce and Giraph for operator implementations  Too much complexity  Performance loss through serialization in HDFS/HBase  0.0.2 Using Flink as execution layer (June 2015)  Basic operators  Currently 0.0.3-SNAPSHOT  Performance improvements  More operator implementations
Operator implementations (0.0.3-SNAPSHOT) Unary Pattern Matching Collection Selection Algorithms LabelPropagation Aggregation Distinct BTG Extraction Projection Sort by FSM Summarization Top Binary Combination Union Overlap Intersection Exclusion Difference Isomorphism Auxiliary Apply Reduce Call
Future Work  Operator integration into Gelly  Summarization FLINK-2411  Graph Sampling  …  Graph Operations on streams (Flink)  Graph Partitioning (maybe together with the Gelly people)  Graph Versioning (Storage)  Benchmarking  GrALa Interpreter / Web UI
Benchmarks Sneak Preview 0 200 400 600 800 1000 1200 1400 1 2 4 8 16 Time [s] # Worker Summarization (Vertex and Edge Labels)  16x Intel(R) Xeon(R) CPU E5-2430 v2 @ 2.50GHz (12 Cores), 48 GB RAM  Hadoop 2.5.2, Flink 0.9.0  slots (per node) 12  jobmanager.heap.mb 2048  taskmanager.heap.mb 40960  Foodbroker Graph (https://github.com/dbs-leipzig/foodbroker)  Generates BI process data  858,624,267 Vertices, 4,406,445,007 Edges, 663GB Payload
Web UI Sneak Preview
Contributions welcome  Code  Operator implementations  Performance Tuning  Storage layout  Data! and Use Cases  We are researchers, we assume ...  Getting real data (especially BI data) is nearly impossible  People  Bachelor / Master / PhD Thesis
Thank you for building Flink! www.gradoop.com https://github.com/dbs-leipzig/gradoop http://dbs.uni-leipzig.de/file/GradoopTR.pdf http://dbs.uni-leipzig.de/file/biiig-vldb2014.pdf

More Related Content

PDF
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
PDF
Batch and Stream Graph Processing with Apache Flink
byVasia Kalavri
 
PPTX
Suneel Marthi - Deep Learning with Apache Flink and DL4J
byFlink Forward
 
PPTX
Apache Flink - Hadoop MapReduce Compatibility
byFabian Hueske
 
PDF
Suneel Marthi – BigPetStore Flink: A Comprehensive Blueprint for Apache Flink
byFlink Forward
 
PDF
Flink Apachecon Presentation
byGyula Fóra
 
PDF
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
byFlink Forward
 
PPTX
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
byFlink Forward
 
Introduction to Apache Flink - Fast and reliable big data processing
byTill Rohrmann
 
Batch and Stream Graph Processing with Apache Flink
byVasia Kalavri
 
Suneel Marthi - Deep Learning with Apache Flink and DL4J
byFlink Forward
 
Apache Flink - Hadoop MapReduce Compatibility
byFabian Hueske
 
Suneel Marthi – BigPetStore Flink: A Comprehensive Blueprint for Apache Flink
byFlink Forward
 
Flink Apachecon Presentation
byGyula Fóra
 
Albert Bifet – Apache Samoa: Mining Big Data Streams with Apache Flink
byFlink Forward
 
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
byFlink Forward
 

What's hot

PPTX
Google cloud Dataflow & Apache Flink
byIván Fernández Perea
 
PDF
Flink Gelly - Karlsruhe - June 2015
byAndra Lungu
 
PDF
Understanding Query Plans and Spark UIs
byDatabricks
 
PDF
Building Robust ETL Pipelines with Apache Spark
byDatabricks
 
PPTX
Apache Flink - Overview and Use cases of a Distributed Dataflow System (at pr...
byStephan Ewen
 
PPTX
ApacheCon: Apache Flink - Fast and Reliable Large-Scale Data Processing
byFabian Hueske
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
PPTX
Intro to Spark development
bySpark Summit
 
PDF
Apache Spark vs Apache Flink
byAKASH SIHAG
 
PDF
Exceptions are the Norm: Dealing with Bad Actors in ETL
byDatabricks
 
PPTX
Robust and Scalable ETL over Cloud Storage with Apache Spark
byDatabricks
 
PDF
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
byFlink Forward
 
PDF
Designing Distributed Machine Learning on Apache Spark
byDatabricks
 
PDF
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
byVasia Kalavri
 
PDF
From Pipelines to Refineries: Scaling Big Data Applications
byDatabricks
 
PDF
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
byVasia Kalavri
 
PDF
Large-Scale Stream Processing in the Hadoop Ecosystem - Hadoop Summit 2016
byGyula Fóra
 
PDF
Maximilian Michels – Google Cloud Dataflow on Top of Apache Flink
byFlink Forward
 
PDF
Scaling Up AI Research to Production with PyTorch and MLFlow
byDatabricks
 
PDF
Fabian Hueske – Juggling with Bits and Bytes
byFlink Forward
 
Google cloud Dataflow & Apache Flink
byIván Fernández Perea
 
Flink Gelly - Karlsruhe - June 2015
byAndra Lungu
 
Understanding Query Plans and Spark UIs
byDatabricks
 
Building Robust ETL Pipelines with Apache Spark
byDatabricks
 
Apache Flink - Overview and Use cases of a Distributed Dataflow System (at pr...
byStephan Ewen
 
ApacheCon: Apache Flink - Fast and Reliable Large-Scale Data Processing
byFabian Hueske
 
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
Intro to Spark development
bySpark Summit
 
Apache Spark vs Apache Flink
byAKASH SIHAG
 
Exceptions are the Norm: Dealing with Bad Actors in ETL
byDatabricks
 
Robust and Scalable ETL over Cloud Storage with Apache Spark
byDatabricks
 
Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka
byFlink Forward
 
Designing Distributed Machine Learning on Apache Spark
byDatabricks
 
Large-scale graph processing with Apache Flink @GraphDevroom FOSDEM'15
byVasia Kalavri
 
From Pipelines to Refineries: Scaling Big Data Applications
byDatabricks
 
Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink
byVasia Kalavri
 
Large-Scale Stream Processing in the Hadoop Ecosystem - Hadoop Summit 2016
byGyula Fóra
 
Maximilian Michels – Google Cloud Dataflow on Top of Apache Flink
byFlink Forward
 
Scaling Up AI Research to Production with PyTorch and MLFlow
byDatabricks
 
Fabian Hueske – Juggling with Bits and Bytes
byFlink Forward
 

Viewers also liked

PPTX
Apache Flink Training: System Overview
byFlink Forward
 
PPTX
Fabian Hueske – Cascading on Flink
byFlink Forward
 
PDF
Mikio Braun – Data flow vs. procedural programming
byFlink Forward
 
PDF
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
byFlink Forward
 
PDF
Anwar Rizal – Streaming & Parallel Decision Tree in Flink
byFlink Forward
 
PDF
Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...
byFlink Forward
 
PDF
Jim Dowling – Interactive Flink analytics with HopsWorks and Zeppelin
byFlink Forward
 
PPTX
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
PPTX
Slim Baltagi – Flink vs. Spark
byFlink Forward
 
PDF
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
byFlink Forward
 
PPTX
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
byFlink Forward
 
PPTX
Apache Flink Training: DataStream API Part 1 Basic
byFlink Forward
 
PPTX
Flink Case Study: Bouygues Telecom
byFlink Forward
 
PDF
Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose
byFlink Forward
 
PDF
Vasia Kalavri – Training: Gelly School
byFlink Forward
 
PDF
Ufuc Celebi – Stream & Batch Processing in one System
byFlink Forward
 
PPTX
Flink 0.10 @ Bay Area Meetup (October 2015)
byStephan Ewen
 
PPTX
Assaf Araki – Real Time Analytics at Scale
byFlink Forward
 
PDF
Marton Balassi – Stateful Stream Processing
byFlink Forward
 
PDF
Matthias J. Sax – A Tale of Squirrels and Storms
byFlink Forward
 
Apache Flink Training: System Overview
byFlink Forward
 
Fabian Hueske – Cascading on Flink
byFlink Forward
 
Mikio Braun – Data flow vs. procedural programming
byFlink Forward
 
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
byFlink Forward
 
Anwar Rizal – Streaming & Parallel Decision Tree in Flink
byFlink Forward
 
Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...
byFlink Forward
 
Jim Dowling – Interactive Flink analytics with HopsWorks and Zeppelin
byFlink Forward
 
Apache Flink Training: DataStream API Part 2 Advanced
byFlink Forward
 
Slim Baltagi – Flink vs. Spark
byFlink Forward
 
Sebastian Schelter – Distributed Machine Learing with the Samsara DSL
byFlink Forward
 
Till Rohrmann – Fault Tolerance and Job Recovery in Apache Flink
byFlink Forward
 
Apache Flink Training: DataStream API Part 1 Basic
byFlink Forward
 
Flink Case Study: Bouygues Telecom
byFlink Forward
 
Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose
byFlink Forward
 
Vasia Kalavri – Training: Gelly School
byFlink Forward
 
Ufuc Celebi – Stream & Batch Processing in one System
byFlink Forward
 
Flink 0.10 @ Bay Area Meetup (October 2015)
byStephan Ewen
 
Assaf Araki – Real Time Analytics at Scale
byFlink Forward
 
Marton Balassi – Stateful Stream Processing
byFlink Forward
 
Matthias J. Sax – A Tale of Squirrels and Storms
byFlink Forward
 

Similar to Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

PPTX
Graphs in data structures are non-linear data structures made up of a finite ...
bybhargavi804095
 
PDF
Distributed graph processing
byBartosz Konieczny
 
PDF
An excursion into Graph Analytics with Apache Spark GraphX
byKrishna Sankar
 
PPTX
Web-Scale Graph Analytics with Apache Spark with Tim Hunter
byDatabricks
 
PPTX
Big data week 2018 - Graph Analytics on Big Data
byChristos Hadjinikolis
 
PDF
Distributed Graph Analytics with Gradoop
byMartin Junghanns
 
PDF
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink & Neo4j Meetup Be...
byMartin Junghanns
 
PDF
Big dataintegration rahm-part3Scalable and privacy-preserving data integratio...
byErhardRahm
 
PDF
Benchmarking tool for graph algorithms
byYash Khandelwal
 
PDF
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
bySpark Summit
 
PDF
GraphTech Ecosystem - part 2: Graph Analytics
byLinkurious
 
PDF
BIG GRAPH: TOOLS, TECHNIQUES, ISSUES, CHALLENGES AND FUTURE DIRECTIONS
bycscpconf
 
PDF
Big Graph : Tools, Techniques, Issues, Challenges and Future Directions
bycsandit
 
PDF
Graph Analytics in Spark
byPaco Nathan
 
PDF
GraphX: Graph analytics for insights about developer communities
byPaco Nathan
 
PDF
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
PDF
Employing Graph Databases as a Standardization Model towards Addressing Heter...
byDippy Aggarwal
 
PDF
Meetup Big Data User Group Dresden: Gradoop - Scalable Graph Analytics with A...
byMartin Junghanns
 
PDF
Gradoop: Scalable Graph Analytics with Apache Flink @ FOSDEM 2016
byMartin Junghanns
 
PDF
Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridz...
byProvectus
 
Graphs in data structures are non-linear data structures made up of a finite ...
bybhargavi804095
 
Distributed graph processing
byBartosz Konieczny
 
An excursion into Graph Analytics with Apache Spark GraphX
byKrishna Sankar
 
Web-Scale Graph Analytics with Apache Spark with Tim Hunter
byDatabricks
 
Big data week 2018 - Graph Analytics on Big Data
byChristos Hadjinikolis
 
Distributed Graph Analytics with Gradoop
byMartin Junghanns
 
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink & Neo4j Meetup Be...
byMartin Junghanns
 
Big dataintegration rahm-part3Scalable and privacy-preserving data integratio...
byErhardRahm
 
Benchmarking tool for graph algorithms
byYash Khandelwal
 
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
bySpark Summit
 
GraphTech Ecosystem - part 2: Graph Analytics
byLinkurious
 
BIG GRAPH: TOOLS, TECHNIQUES, ISSUES, CHALLENGES AND FUTURE DIRECTIONS
bycscpconf
 
Big Graph : Tools, Techniques, Issues, Challenges and Future Directions
bycsandit
 
Graph Analytics in Spark
byPaco Nathan
 
GraphX: Graph analytics for insights about developer communities
byPaco Nathan
 
Web-Scale Graph Analytics with Apache® Spark™
byDatabricks
 
Employing Graph Databases as a Standardization Model towards Addressing Heter...
byDippy Aggarwal
 
Meetup Big Data User Group Dresden: Gradoop - Scalable Graph Analytics with A...
byMartin Junghanns
 
Gradoop: Scalable Graph Analytics with Apache Flink @ FOSDEM 2016
byMartin Junghanns
 
Data Summer Conf 2018, “Analysing Billion Node Graphs (ENG)” — Giorgi Jvaridz...
byProvectus
 

More from Flink Forward

PPTX
Apache Flink in the Cloud-Native Era
byFlink Forward
 
PDF
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
byFlink Forward
 
PDF
Building a fully managed stream processing platform on Flink at scale for Lin...
byFlink Forward
 
PDF
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
byFlink Forward
 
PDF
Batch Processing at Scale with Flink & Iceberg
byFlink Forward
 
PPTX
Dynamic Rule-based Real-time Market Data Alerts
byFlink Forward
 
PDF
Flink powered stream processing platform at Pinterest
byFlink Forward
 
PPTX
The Current State of Table API in 2022
byFlink Forward
 
PDF
Tame the small files problem and optimize data layout for streaming ingestion...
byFlink Forward
 
PPTX
Autoscaling Flink with Reactive Mode
byFlink Forward
 
PDF
Introducing the Apache Flink Kubernetes Operator
byFlink Forward
 
PPTX
Evening out the uneven: dealing with skew in Flink
byFlink Forward
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
PPTX
Where is my bottleneck? Performance troubleshooting in Flink
byFlink Forward
 
PPTX
Tuning Apache Kafka Connectors for Flink.pptx
byFlink Forward
 
PDF
Flink SQL on Pulsar made easy
byFlink Forward
 
PPTX
Using the New Apache Flink Kubernetes Operator in a Production Deployment
byFlink Forward
 
PPTX
Processing Semantically-Ordered Streams in Financial Services
byFlink Forward
 
PPTX
One sink to rule them all: Introducing the new Async Sink
byFlink Forward
 
PPTX
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
byFlink Forward
 
Apache Flink in the Cloud-Native Era
byFlink Forward
 
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
byFlink Forward
 
Building a fully managed stream processing platform on Flink at scale for Lin...
byFlink Forward
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
byFlink Forward
 
Batch Processing at Scale with Flink & Iceberg
byFlink Forward
 
Dynamic Rule-based Real-time Market Data Alerts
byFlink Forward
 
Flink powered stream processing platform at Pinterest
byFlink Forward
 
The Current State of Table API in 2022
byFlink Forward
 
Tame the small files problem and optimize data layout for streaming ingestion...
byFlink Forward
 
Autoscaling Flink with Reactive Mode
byFlink Forward
 
Introducing the Apache Flink Kubernetes Operator
byFlink Forward
 
Evening out the uneven: dealing with skew in Flink
byFlink Forward
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
byFlink Forward
 
Where is my bottleneck? Performance troubleshooting in Flink
byFlink Forward
 
Tuning Apache Kafka Connectors for Flink.pptx
byFlink Forward
 
Flink SQL on Pulsar made easy
byFlink Forward
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
byFlink Forward
 
Processing Semantically-Ordered Streams in Financial Services
byFlink Forward
 
One sink to rule them all: Introducing the new Async Sink
byFlink Forward
 
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
byFlink Forward
 

Recently uploaded

PDF
Tune System Performance - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Build Agentic AI Applications with Oracle AI Database Private Agent Factory
bySandesh Rao
 
PDF
Flatpak CLI Application Management & Building on RHEL.pdf
byLinuxCert Guru
 
PDF
#MakeAIMatter for HR Professionals | AI Transformation Workshop by Tekdi Tech...
byParth Lawate
 
PPTX
Governance, Deployment & Methodologies for Agentic Automation [2/3]
bysuhanisingh58689
 
PDF
Inclusive India_Samir_Dash_10 NOV_FINAL 2025.pdf
bySamir Dash
 
PDF
Implement Advanced Storage in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Dr. Robert Krug - Specializes In Predictive Modeling
byDr. Robert Krug
 
PDF
Schedule Future Tasks - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Upskill to Agentic Automation - Accelerating Your Job Search using AI
byDianaGray10
 
PPTX
Career Blueprint - Future Career Vision & Success Stories - 2025 - Part 1
byDianaGray10
 
PDF
Access Network Attached Storage in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
Manage Network Security (Firewall) in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
PDF
ENTSO-E's Response to the European Commission Call for Evidence on the Strate...
byReynir Orn Bachmann Gudmundsson
 
PDF
Reset RHEL Root User Password - RHCSA.pdf
byLinuxCert Guru
 
PDF
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
PPTX
Standardising and simplifying systems and data management - Esri UK Welsh Con...
byEsri UK
 
PDF
Webinar: Introduction to LF Energy SEAPATH
byDanBrown980551
 
PDF
Case Study: How LKQ Corporation Improved Training Efficiency & Delivery
byRustici Software
 
PPTX
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
Tune System Performance - RHCSA (RH134).pdf
byLinuxCert Guru
 
Build Agentic AI Applications with Oracle AI Database Private Agent Factory
bySandesh Rao
 
Flatpak CLI Application Management & Building on RHEL.pdf
byLinuxCert Guru
 
#MakeAIMatter for HR Professionals | AI Transformation Workshop by Tekdi Tech...
byParth Lawate
 
Governance, Deployment & Methodologies for Agentic Automation [2/3]
bysuhanisingh58689
 
Inclusive India_Samir_Dash_10 NOV_FINAL 2025.pdf
bySamir Dash
 
Implement Advanced Storage in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
Dr. Robert Krug - Specializes In Predictive Modeling
byDr. Robert Krug
 
Schedule Future Tasks - RHCSA (RH134).pdf
byLinuxCert Guru
 
Upskill to Agentic Automation - Accelerating Your Job Search using AI
byDianaGray10
 
Career Blueprint - Future Career Vision & Success Stories - 2025 - Part 1
byDianaGray10
 
Access Network Attached Storage in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
Manage Network Security (Firewall) in RHEL - RHCSA (RH134).pdf
byLinuxCert Guru
 
ENTSO-E's Response to the European Commission Call for Evidence on the Strate...
byReynir Orn Bachmann Gudmundsson
 
Reset RHEL Root User Password - RHCSA.pdf
byLinuxCert Guru
 
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
Standardising and simplifying systems and data management - Esri UK Welsh Con...
byEsri UK
 
Webinar: Introduction to LF Energy SEAPATH
byDanBrown980551
 
Case Study: How LKQ Corporation Improved Training Efficiency & Delivery
byRustici Software
 
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 

Martin Junghans – Gradoop: Scalable Graph Analytics with Apache Flink

  • 1.
    GRADOOP: Scalable Graph Analyticswith Apache Flink Martin Junghanns University of Leipzig
  • 2.
    About the speakerand the team  2011 Bachelor of Engineering  Thesis: Partitioning of Dynamic Graphs  2014 Master of Science  Thesis: Graph Database Systems for Business Intelligence  Now: PhD Student, Database Group, University of Leipzig  Distributed Systems  Distributed Graph Data Management  Graph Theory & Algorithms  Professional Experience: sones GraphDB, SAP André, PhD Student Martin, PhD Student Kevin, M.Sc. StudentNiklas, M.Sc. Student
  • 3.
  • 4.
    𝑮𝑟𝑟𝑟𝑟 = (𝑽𝑒𝑒𝑒𝑒𝑒𝑒𝑒,𝑬𝑑𝑑𝑑𝑑) “Graphs are everywhere”
  • 5.
    𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔,𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
  • 6.
    𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔,𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
  • 7.
    𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔,𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
  • 8.
    𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔,𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy
  • 9.
    𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔,𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy Trent
  • 10.
    𝐺𝐺𝐺𝐺𝐺 = (𝐔𝐔𝐔𝐔𝐔,𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹) “Graphs are everywhere” Alice Bob Eve Dave Carol Mallory Peggy Trent
  • 11.
    𝐺𝐺𝐺𝐺𝐺 = (𝐂𝐂𝐂𝐂𝐂𝐂,𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶) “Graphs are everywhere” Leipzig pop: 544K Dresden pop: 536K Berlin pop: 3.5M Hamburg pop: 1.7M Munich pop: 1.4M Chemnitz pop: 243K Nuremberg pop: 500K Cologne pop: 1M
  • 12.
     World WideWeb  ca. 1 billion websites “Graphs are large”  Facebook  ca. 1.49 billion active users  ca. 340 friends per user
  • 13.
    End-to-End Graph Analytics DataIntegration Graph Analytics Representation
  • 14.
    End-to-End Graph Analytics DataIntegration Graph Analytics Representation  Integrate data from one or more sources into a dedicated graph storage with common graph data model
  • 15.
    End-to-End Graph Analytics DataIntegration Graph Analytics Representation  Integrate data from one or more sources into a dedicated graph storage with common graph data model  Definition of analytical workflows from operator algebra
  • 16.
    End-to-End Graph Analytics DataIntegration Graph Analytics Representation  Integrate data from one or more sources into a dedicated graph storage with common graph data model  Definition of analytical workflows from operator algebra  Result representation in a meaningful way
  • 17.
    Graph Data Management GraphDatabase Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
  • 18.
    Graph Data Management GraphDatabase Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
  • 19.
    Graph Data Management GraphDatabase Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
  • 20.
    Graph Data Management GraphDatabase Systems Neo4j, OrientDB Graph Processing Systems Pregel, Giraph Distributed Workflow Systems Flink Gelly, Spark GraphX Data Model Rich Graph Models Generic Graph Models Generic Graph Models Focus Local ACID Operations Global Graph Operations Global Data and Graph Operations Query Language Yes No No Persistency Yes No No Scalability Vertical Horizontal Horizontal Workflows No No Yes Data Integration No No No Graph Analytics No Yes Yes Representation Yes No No
  • 21.
    What‘s missing? An end-to-endframework and research platform for efficient, distributed and domain independent graph data management and analytics.
  • 22.
    What‘s missing? An end-to-endframework and research platform for efficient, distributed and domain independent graph data management and analytics.
  • 23.
  • 24.
    High Level Architecture HDFS/YARN Cluster HBaseDistributed Graph Store Extended Property Graph Model Flink Operator Implementations Data Integration Flink Operator Execution Workflow Declaration Visual GrALa DSL Representation Data flow Control flow Graph Analytics Representation Workflow Execution
  • 25.
    High Level Architecture HBaseDistributed Graph Store Extended Property Graph Model Flink Operator Implementations Data Integration Flink Operator Execution Workflow Declaration Visual GrALa DSL Representation Data flow Control flow Graph Analytics Representation Workflow Execution HDFS/YARN Cluster
  • 26.
  • 27.
  • 28.
  • 29.
    Graph Operators Operator GrALanotation Binary Combination graph.combine(otherGraph) : Graph Overlap graph.overlap(otherGraph) : Graph Exclusion graph.exclude(otherGraph) : Graph Isomorphism graph.isIsomorphicTo(otherGraph) : Boolean Unary Pattern Matching graph.match(patternGraph,predicate) : Collection Aggregation graph.aggregate(propertyKey,aggregateFunction) : Graph Projection graph.project(vertexFunction,edgeFunction) : Graph Summarization graph.summarize( vertexGroupKeys,vertexAggregateFunction, edgeGroupKeys,edgeAggregateFunction) : Graph
  • 30.
    Combination 1: personGraph =db.G[0].combine(db.G[1]).combine(db.G[2])
  • 31.
    Combination 1: personGraph =db.G[0].combine(db.G[1]).combine(db.G[2])
  • 32.
    Graph Operators Operator GrALanotation Binary Combination graph.combine(otherGraph) : Graph Overlap graph.overlap(otherGraph) : Graph Exclusion graph.exclude(otherGraph) : Graph Isomorphism graph.isIsomorphicTo(otherGraph) : Boolean Unary Pattern Matching graph.match(patternGraph,predicate) : Collection Aggregation graph.aggregate(propertyKey,aggregateFunction) : Graph Projection graph.project(vertexFunction,edgeFunction) : Graph Summarization graph.summarize( vertexGroupKeys,vertexAggregateFunction, edgeGroupKeys,edgeAggregateFunction) : Graph
  • 33.
    Summarization 1: personGraph =db.G[0].combine(db.G[1]).combine(db.G[2]) 2: vertexGroupingKeys = {:type, “city”} 3: edgeGroupingKeys = {:type} 4: vertexAggFunc = (Vertex vSum, Set vertices => vSum[“count”] = |vertices|) 5: edgeAggFunc = (Edge eSum, Set edges => eSum[“count”] = |edges|) 6: sumGraph = personGraph.summarize(vertexGroupingKeys, vertexAggFunc, edgeGroupingKeys, edgeAggFunc)
  • 34.
    Summarization 1: personGraph =db.G[0].combine(db.G[1]).combine(db.G[2]) 2: vertexGroupingKeys = {:type, “city”} 3: edgeGroupingKeys = {:type} 4: vertexAggFunc = (Vertex vSum, Set vertices => vSum[“count”] = |vertices|) 5: edgeAggFunc = (Edge eSum, Set edges => eSum[“count”] = |edges|) 6: sumGraph = personGraph.summarize(vertexGroupingKeys, vertexAggFunc, edgeGroupingKeys, edgeAggFunc)
  • 35.
    Graph Collection Operators OperatorGrALa notation Collection Selection collection.select(predicate) : Collection Distinct collection.distinct() : Collection Sort by collection.sortBy(key, [:asc|:desc]) : Collection Top collection.top(limit) : Collection Union collection.union(otherCollection) : Collection Intersection collection.intersect(otherCollection) : Collection Difference collection.difference(otherCollection) : Collection Auxiliary Apply collection.apply(unaryGraphOperator) : Collection Reduce collection.reduce(binaryGraphOperator) : Graph Call [graph|collection].callFor[Graph|Collection]( algorithm,parameters) : [Graph|Collection]
  • 36.
    Selection 1: collection =<db.G[0],db.G[1],db.G[2]> 2: predicate = (Graph g => |g.V| > 3) 3: result = collection.select(predicate)
  • 37.
    Selection 1: collection =<db.G[0],db.G[1],db.G[2]> 2: predicate = (Graph g => |g.V| > 3) 3: result = collection.select(predicate)
  • 38.
    Graph Collection Operators OperatorGrALa notation Collection Selection collection.select(predicate) : Collection Distinct collection.distinct() : Collection Sort by collection.sortBy(key, [:asc|:desc]) : Collection Top collection.top(limit) : Collection Union collection.union(otherCollection) : Collection Intersection collection.intersect(otherCollection) : Collection Difference collection.difference(otherCollection) : Collection Auxiliary Apply collection.apply(unaryGraphOperator) : Collection Reduce collection.reduce(binaryGraphOperator) : Graph Call [graph|collection].callFor[Graph|Collection]( algorithm,parameters) : [Graph|Collection]
  • 39.
    Extended Property GraphModel in Flink ID Label Properties Graphs ID Label Properties Source Vertex Target Vertex Graphs VertexData EdgeData GraphData ID Label Properties POJO POJO POJO DataSet<Vertex<ID,VertexData>> DataSet<Edge<ID,EdgeData>> DataSet<Subgraph<ID,GraphData>> Gelly 𝒱 ℰ 𝒢 Pojo Representation
  • 40.
    Extended Property GraphModel in Flink VertexData EdgeData GraphData POJO POJO POJO DataSet<Vertex<ID,VertexData>> DataSet<Edge<ID,EdgeData>> DataSet<Subgraph<ID,GraphData>> Gelly VertexData EdgeData GraphData Tuple Tuple Tuple DataSet<VertexData> DataSet<EdgeData> DataSet<GraphData> 𝒱 𝒱 ℰ ℰ 𝒢 𝒢 Pojo Representation Tuple Representation ID Label Properties Graphs ID Label Properties Source Vertex Target Vertex Graphs ID Label Properties ID Label Properties Graphs ID Label Properties Source Vertex Target Vertex Graphs ID Label Properties
  • 41.
    Summarization in Flink VIDCity 0 L 1 L 2 D 3 D 4 D 5 B EID S T 0 0 1 1 1 0 2 1 2 3 2 1 4 2 3 5 3 2 6 4 0 7 4 1 8 5 2 9 5 3 L [0,1] D [2,3,4] B [5] VID City Count 0 L 2 2 D 3 5 B 1 VID Rep 0 0 1 0 2 2 3 2 4 2 5 5 ID S T 0 0 1 1 0 0 2 0 2 3 2 1 4 2 3 5 2 2 6 2 0 7 2 1 8 5 2 9 5 3 ID S T 0 0 0 1 0 0 2 0 2 3 2 0 4 2 2 5 2 2 6 2 0 7 2 0 8 5 2 9 5 2 0,0 [0,1] 0,2 [2] 2,0 [3,6,7] 2,2 [4,5] 5,2 [8,9] EID S T Count 0 0 1 2 2 0 2 1 3 2 0 3 4 2 2 2 8 5 2 2 join(VID==S) 𝒱 ℰ’ 𝒱′ ℰ groupBy(City) reduceGroup + filter + map reduceGroup + filter + map groupBy(S,T) join(VID==T)
  • 42.
    Use Case: GraphBusiness Intelligence
  • 43.
    Use Case: GraphBusiness Intelligence  Business intelligence usually based on relational data warehouses  Enterprise data is integrated within dimensional schema  Analysis limited to predefined relationships  No support for relationship-oriented data mining Facts Dim 1 Dim 2 Dim 3
  • 44.
    Use Case: GraphBusiness Intelligence  Business intelligence usually based on relational data warehouses  Enterprise data is integrated within dimensional schema  Analysis limited to predefined relationships  No support for relationship-oriented data mining  Graph-based approach  Integrate data sources within an instance graph by preserving original relationships between data objects (transactional and master data)  Determine subgraphs (business transaction graphs) related to business activities  Analyze subgraphs or entire graphs with aggregation queries, mining relationship patterns, etc. Facts Dim 1 Dim 2 Dim 3
  • 45.
  • 46.
    Business Transaction Graphs CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy basedOn serves serves bills bills bills processedBy
  • 47.
    Business Transaction Graphs CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 48.
    Business Transaction Graphs CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 49.
    Business Transaction Graphs CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 50.
    Business Transaction Graphs CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 51.
    Business Transaction Graphs CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 52.
    BTG 1 (1) BTGExtraction BTG 2 BTG 3 BTG 4 BTG 5 BTG n …
  • 53.
    (1) BTG Extraction //generate base collection btgs = iig.callForCollection( :BusinessTransactionGraphs , {} )
  • 54.
    (2) Profit Aggregation CITERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 55.
    (2) Profit Aggregation //generate base collection btgs = iig.callForCollection( :BusinessTransactionGraphs , {} ) // define profit aggregate function aggFunc = ( Graph g => g.V.values(“Revenue").sum() - g.V.values(“Expense").sum() )
  • 56.
    (2) Profit Aggregation BTG1 BTG 2 BTG 3 BTG 4 BTG 5 BTG n … ∑ Revenue ∑ Expenses Net Profit 5,000 -3,000 2,000 9,000 -3,000 6,000 2,000 -1,500 500 5,000 -7,000 -2,000 10,000 -15,000 -5,000 … … … 8,000 -4,000 4,000
  • 57.
    (2) Profit Aggregation //generate base collection btgs = iig.callForCollection( :BusinessTransactionGraphs , {} ) // define profit aggregate function aggFunc = ( Graph g => g.V.values(“Revenue").sum() - g.V.values(“Expense").sum() ) // apply aggregate function and store result at new property btgs = btgs.apply( Graph g => g.aggregate( “Profit“ , aggFunc ) )
  • 58.
    (3) BTG Clustering BTG1 BTG 2 BTG 3 BTG 4 BTG 5 BTG n … ∑ Revenue ∑ Expenses Net Profit 5,000 -3,000 2,000 9,000 -3,000 6,000 2,000 -1,500 500 5,000 -7,000 -2,000 10,000 -15,000 -5,000 … … … 8,000 -4,000 4,000
  • 59.
    (3) BTG Clustering //select profit and loss clusters profitBtgs = btgs.select( Graph g => g[“Profit”] >= 0 ) lossBtgs = btgs.difference(profitBtgs)
  • 60.
    (4) Cluster CharacteristicPatterns CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 61.
    (4) Cluster CharacteristicPatterns CIT ERP Employee Name: Dave Employee Name: Alice Employee Name: Bob Employee Name: Carol Ticket Expense: 500 SalesQuotation SalesOrder PurchaseOrder PurchaseOrder SalesRevenue Revenue: 5,000 PurchaseInvoice Expense: 2,000 PurchaseInvoice Expense: 1,500 sentBy createdBy processedBy createdBy openedFor processedBy processedBy basedOn serves serves bills bills bills
  • 62.
    (4) Cluster CharacteristicPatterns BTG 1 BTG 2 BTG 3 BTG 4 BTG 5 BTG n … ∑ Revenue ∑ Expenses Net Profit 5,000 -3,000 2,000 9,000 -3,000 6,000 2,000 -1,500 500 5,000 -7,000 -2,000 10,000 -15,000 -5,000 … … … 8,000 -4,000 4,000 TicketAlice processedBy Bob createdBy PurchaseOrder
  • 63.
    (4) Cluster CharacteristicPatterns // select profit and loss clusters profitBtgs = btgs.select( Graph g => g[“Profit”] >= 0 ) lossBtgs = btgs.difference(profitBtgs) // apply magic profitFreqPats = profitBtgs.callForCollection( :FrequentSubgraphs , {“Threshold”:0.7} ) lossFreqPats = lossBtgs.callForCollection( :FrequentSubgraphs , {“Threshold”:0.7} ) // determine cluster characteristic patterns trivialPats = profitFreqPats.intersect(lossFreqPats) profitCharPatterns = profitFreqPats.difference(trivialPats) lossCharPatterns = lossFreqPats.difference(trivialPats)
  • 64.
    Current State &Future Work
  • 65.
    Current State  0.0.1First Prototype (May 2015)  Hadoop MapReduce and Giraph for operator implementations  Too much complexity  Performance loss through serialization in HDFS/HBase  0.0.2 Using Flink as execution layer (June 2015)  Basic operators  Currently 0.0.3-SNAPSHOT  Performance improvements  More operator implementations
  • 66.
    Operator implementations (0.0.3-SNAPSHOT) UnaryPattern Matching Collection Selection Algorithms LabelPropagation Aggregation Distinct BTG Extraction Projection Sort by FSM Summarization Top Binary Combination Union Overlap Intersection Exclusion Difference Isomorphism Auxiliary Apply Reduce Call
  • 67.
    Future Work  Operatorintegration into Gelly  Summarization FLINK-2411  Graph Sampling  …  Graph Operations on streams (Flink)  Graph Partitioning (maybe together with the Gelly people)  Graph Versioning (Storage)  Benchmarking  GrALa Interpreter / Web UI
  • 68.
    Benchmarks Sneak Preview 0 200 400 600 800 1000 1200 1400 12 4 8 16 Time [s] # Worker Summarization (Vertex and Edge Labels)  16x Intel(R) Xeon(R) CPU E5-2430 v2 @ 2.50GHz (12 Cores), 48 GB RAM  Hadoop 2.5.2, Flink 0.9.0  slots (per node) 12  jobmanager.heap.mb 2048  taskmanager.heap.mb 40960  Foodbroker Graph (https://github.com/dbs-leipzig/foodbroker)  Generates BI process data  858,624,267 Vertices, 4,406,445,007 Edges, 663GB Payload
  • 69.
    Web UI SneakPreview
  • 70.
    Contributions welcome  Code Operator implementations  Performance Tuning  Storage layout  Data! and Use Cases  We are researchers, we assume ...  Getting real data (especially BI data) is nearly impossible  People  Bachelor / Master / PhD Thesis
  • 71.
    Thank you forbuilding Flink! www.gradoop.com https://github.com/dbs-leipzig/gradoop http://dbs.uni-leipzig.de/file/GradoopTR.pdf http://dbs.uni-leipzig.de/file/biiig-vldb2014.pdf