Accumulo Summit 2014: Accumulo backed Tinkerpop Implementation

TinkerPop Backed By Accumulo
6/12/2014
Ryan Webb
Associate Professional
Ryan.Webb@jhuapl.edu

Agenda
 Introduction to TinkerPop
 Detailed Implementation
 Obstacles
 Overcoming Obstacles
 Map Reduce Integration
 Performance

Background
 Associate Professional at The Johns Hopkins Applied Physics
Laboratory
 Bachelors of Science in Computer Science with a minor in
Mathematics from the University of Delaware
 Pursing a Masters in Computer Science with a focus on
Distributed Systems at the Whiting School of Engineering

TinkerPop Blueprints
 Foundational technology for a
complete graph stack
 Extensive test suite to ensure
implementations follow all the
rules required.
 Only a simple API
 getVertex
 getEdge
 setProperty
 getProperty
 Multiple Interfaces with
incremental features

TinkerPop Blueprints Graph API

Graph Creation
Configuration cfg = new AccumuloGraphConfiguration()
.instance("accumulo").user("user").zkHosts("zk1")
.password("password".getBytes()).name("myGraph");
Graph graph = GraphFactory.open(cfg);
Vertex v1 = graph.addVertex("1");
v1.setProperty("name", "Alice");
Vertex v2 = graph.addVertex("2");
v2.setProperty("name", "Bob");
Edge e1 = graph.addEdge("E1", v1, v2, "knows");
e1.setProperty("since", new Date());

Trade off Spectrum
Consistency
Performance

Accumulo Implementation
 Base Naïve implementation passes all required TinkerPop tests
 Far Right of the spectrum
 As consistent as you can get
 Table Structure
 Edge and Vertex
 Edge and Vertex Index table
 Metadata Table for indexes

Table Structure
Vertex
Edge
Row ID Column Family Column Qualifier Value
VertexID Label Flag Exists Flag [empty]
VertexID INVERTEX OutVertexID_EdgeID Edge Label
VertexID OUTVERTEX InVertexID_EdgeID Edge Label
VertexID Property Key [empty] Serialized Value
Row ID Column Family Column Qualifier Value
EdgeID Label Flag InVertexID_OutVertexID Edge Label
EdgeID Property Key [empty] Serialized Value

Graph Access and Index Creation/Use
// Access before Index
for (Vertex v: graph.getVertices()) {
String name = v.getProperty("name");
}
((KeyIndexableGraph)graph)
.createKeyIndex("name", Vertex.class);
// Access after Index
for (Vertex v: graph.getVertices()) {
String name = v.getProperty("name");
}

Table Structure - Continued
Indexes
Metadata
Row Column Family Column Qualifier Value
Serialized Value Property Key VertexID [empty]
Row Column Family Column Qualifier Value
Index Name Index Class [empty] [empty]

Obstacles
 Existence checking is expensive
 Required for TinkerPop test suite
 Writing every graph object out is expensive
 Building indexes post ingest is expensive
 Blocking, full table scan
 Consistency is expensive

Overcoming Obstacles
Give more power to users who know they are using an Accumulo
Graph
 Ingest Improvements
 Give option to disable existence checks
 Allow manual batching
 Specialized Ingest path
 Traversal Improvements
 Attribute preloading
 Property caching
 Element caching

Simple Bulk Ingest
// Will migrate to BatchGraph
AccumuloBulkIngester g = new AccumuloBulkIngester(cfg);
PropertyBuilder v1 = g.addVertex("ID1");
PropertyBuilder v2 = g.addVertex("ID2");
PropertyBuilder edge = g.addEdge("ID1", "ID2", "knows");
v1.add("name", "alice");
v2.add("name", "bob");
edge.add("since", new Date());

Map Reduce Integration
 In your Tool
j.setInputFormatClass(VertexInputFormat.class);
VertexInputFormat.setAccumuloGraphConfiguration(
new AccumuloGraphConfiguration()
.instance(“accumulo").zkHosts(“zk1").user("root")
.password(“secret".getBytes()).name("myGraph"));
 In your Mapper
public void map(Text k, Vertex v, Context c){
System.out.println(v.getId().toString());
}

Results
2 Nodes 4 Nodes 8 Nodes
20 Hours 9 Minutes 13 Hours 47 Minutes 7 Hours 4 Minutes
Cluster Stats
8 Node Cluster
64 GB Ram
Quad-Core Xeon Processor 2.50GHz 10MB
2x 4 TB 6.0Gb/s 7200 RPM Drives
1 Gb/s Networking
Accumulo 1.5.1, Hadoop 2.0.0 – MR1
Stanford SNAP Friendster Graph
65,608,366 Vertices
1,806,067,135 Edges
2 Nodes 4 Nodes 8 Nodes
55 Minutes 29 Minutes 15 Minutes
Vertex Iteration
Ingest

Conclusion
 Simple, easy to read graph API
 Give developers a lot of tuning points for their implementations
 Performance is “good enough”
 Not meant for high performance, specialized solutions
 Quick to develop new ideas and investigate your graph.
 Easy to integrate and already integrated.
 Low effort to get REST access to your graph

Future
 Polish and open source
 Iterators
 Locality Groups
 Addressing Security
 Graph Query
 Extending MapReduce Integration
 Upgrading to Accumulo 1.6, TinkerPop 2.5
 Conditional Mutations
 Table namespaces

Resources
 http://www.tinkerpop.com/
 http://snap.stanford.edu/data/com-Friendster.html

Accumulo Summit 2014: Accumulo backed Tinkerpop Implementation

Accumulo Summit 2014: Accumulo backed Tinkerpop Implementation

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Accumulo Summit 2014: Accumulo backed Tinkerpop Implementation (20)

Recently uploaded (20)

Accumulo Summit 2014: Accumulo backed Tinkerpop Implementation