Hadoop application architectures - using Customer 360 as an example

Hadoop Application
Architectures:
Architecting a Next
Generation Data
Platform
Strata + Hadoop World, New York 2016
tiny.cloudera.com/app-arch-ny
tiny.cloudera.com/app-arch-questions
Mark Grover | @mark_grover
Ted Malaska | @ted_malaska
Jonathan Seidman | @jseidman

Questions? tiny.cloudera.com/app-arch-
questions
Logistics
▪ Break at 3:00 – 3:30 PM
▪ Questions at the end of each section
▪ Slides at tiny.cloudera.com/app-arch-ny
▪ Code at https://github.com/hadooparchitecturebook/Taxi360

questions
About the book
▪ @hadooparchbook
▪ hadooparchitecturebook.com
▪ github.com/hadooparchitecturebook
▪ slideshare.com/hadooparchbook

questions
About the presenters
▪ Principal Solutions Architect
at Cloudera
▪ Previously, lead architect at
FINRA
▪ Contributor to Apache
Hadoop, HBase, Flume, Avro,
Pig, Spark, YARN, Sqoop,
Kudu, Kafka
Ted Malaska

questions
▪ Partner Software Engineer at
Cloudera
▪ Contributor to Apache Sqoop.
▪ Previously, Technical Lead on
the big data team at Orbitz,
co-founder of the Chicago
Hadoop User Group and
Chicago Big Data
Jonathan Seidman

questions
▪ Software Engineer on Spark
at Cloudera
▪ Committer on Apache Bigtop,
PMC member on Apache
Sentry(incubating)
▪ Contributor to Apache Spark,
Hadoop, Hive, Sqoop, Pig,
Flume
Mark Grover

Case Study Overview
Internet of Things and Entity 360

questions
Customer 360

questions
Connected Cars

questions
Entity (Taxi) 360 View
Geo-location/
Traffic Data
Customer Data
Maintenance
Data
Other Data
Sources
Streaming
Vehicle Data

questions
What Makes Hadoop a Fit?
Data Sources Extract Transform Load
The early days…

questions
What Makes Hadoop a Fit?
SERVERS MARTS EDWS DOCUMENTS STORAGE SEARCH ARCHIVE
ERP, CRM, RDBMS, MACHINES FILES, IMAGES, VIDEOS, LOGS, CLICKSTREAMS EXTERNAL DATA SOURCES
Today…

questions
Enabling a Range of New Use Cases…
Fraud
Detection
Market
transactions
Internet of
Things
Network
Security

questions
Hadoop Challenges
Kafka StreamsKafka Connect
Kafka

questions
Challenges - Architectural Considerations
▪ Reliable and scalable ingress of multiple data types and sources:
- High volume event data? Batch data?
▪ Reliable and scalable storage to support multiple workloads and access patterns
- Historical data? Real-time search? Analytics
▪ Processing engines (for background processing):
- Stream processing? Batch processing?
▪ Data Modeling
- Modeling data for real-time random access? Analytic access? Batch access?

Case Study
Requirements
Overview

questions
Requirements
▪ Allow users (technical and non-technical) to analyze and visualize data…

questions
Requirements
▪ Provide analysts with query capabilities via a standard interface…

questions
Requirements
▪ Provide developers the ability to perform batch processing on historical data…

questions
Requirements
▪ To support all this, we need:
- Reliable ingestion of streaming and batch data.
- Ability to perform transformations on streaming data in flight.
- Ability to perform sophisticated processing of historical data.

High level architecture
Walkthrough

questions
Source Transport Stream
Processing
Storage Access
Data Producer Pub-Sub
Processing &
Ingestion
Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series
Lookup
Batch
Processing
SQL
NRT REST
NRT
Dashboard

questions
But wait!
What about batch data?

questions
Buffering Data
▪ What do we mean by “buffering” and why do we need it?
event,event,event,event,event,event…
This is bad!

questions
Buffering Data – Message Brokers
Publisher
Publisher
Publisher
Message
Queue
Subscriber
Subscriber
Subscriber

questions
Processing
Storage Access
Custom
Producer
or
Processing &
Ingestion
Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series
Lookup
Batch
Processing
SQL
NRT Rest
NRT
Dashboard

questions
Buffering Data – Flume vs. Kafka
▪ Flume – well integrated with Hadoop.
- Great choice when ingesting data into HDFS.
- Can support simple transformations.
- Less coding.
▪ But…
- Interface between Kafka and the streaming layer is already well defined.
- Transformations are done in the stream processing layer.
- We need a more general purpose system at this layer.

questions
Kafka Connect
Kafka
Connect
(Source)
Kafka
Connect
(Sink)

questions
What is Kafka?
▪ It’s like a message queue, right?
- Actually, it’s a “distributed commit log”.
0 1 2 3 4 5 6 7 8
Data
Source
Data
Consumer
A
Data
Consumer
B

questions
Topics and Partitions
▪ Messages are organized into topics, and each topic is split into partitions.
- Each partition is an immutable, time-sequenced log of messages.
0 1 2 3 4 5 6 7 8
0 1 2 3 4 5 6 7 8
0 1 2 3 4 5 6 7 8
Partition 0
Partition 1
Partition 2
Data
Source
Topi
c

questions
Kafka Background (Physical)
ProducersProducersProducers
BrockerBrockerBroker
ConsumersConsumersConsumers
ZooKeeperZooKeeperZooKeeper

questions
Consumers
Kafka In Our Architecture
Taxi Trip Data
Producer
Kafka
taxi-trip-input
Topic
Stream
Processing
(Analytic)
Stream
Processing
(Lookup)
Stream
Processing
(Search)
Stream
Processing
(Long Term)

questions
Kafka Considerations – Reliability
▪ Different reliability levels for topics:
Taxi Trip Data
Kafka
taxi-trip-input
Twitter customer-sentiment
100% – dups
are ok
(“At least
once”)
<=100%
(“At most
once”)

questions
Kafka Considerations – Reliability
▪ But remember there are tradeoffs…

questions
Kafka Reliability – Replication
Producer
Broker
Partition1
Partition2
Partition3
Leader

questions
Producer
Broker
Partition1
Partition2
Partition3

questions
Producer
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Leader

questions
Kafka Reliability– Replication
Producer
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Leader
Leader

questions
Producer
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Broker
Partition1
Partition2
Partition3
Leader

questions
▪ So how does this relate to our application?
kafka-topics --zookeeper ZKHOST:ZKPORT –partition 2 --replication-factor 3
--create --topic taxi-trip-input
kafka-topics --zookeeper ZKHOST:ZKPORT –partition 2 --replication-factor 1
--create –topic customer-sentiment

questions
Kafka Reliability – Producers
Taxi Trip Data
Kafka
taxi_trip_input
Partition 1
Partition 2
Partition 3
Topic B
Partition 1
Partition 2
Partition 3
Message
failure?
Producer
Resend
message
acks=all

questions
Kafka Reliability – Producers
▪ What about duplicates?
Taxi Trip Data
Kafka
taxi_trip_input
Partition 1
Partition 2
Partition 3
Topic B
Partition 1
Partition 2
Partition 3
Producer
ID Message
1000 2009-01-04 03:02:00,1,2.629,...
1001 2009-01-04 03:38:00,3,4.549…
1001 2009-01-04 03:38:00,3,4.549…

questions
Kafka Scaling – Partitions
Producer
Kafka
taxi-trip-input
Partition 1
Partition 2
Partition 3
Consumer Group
Consumer
Consumer
Consumer

questions
Producer
Kafka
taxi-trip-input
Partition 1
Partition 2
Partition 3
Consumer Group
Consumer
Consumer
Consumer
Partition 4
Partition 5
Consumer
Consumer
Higher
throughput
Higher
throughput
More
resources
(memory)
More
resources
(file handles)
Producer

questions

questions
Kafka Scaling – Producers
Producer
Kafka
taxi-trip-input
Partition 1
Partition 2
Partition 3
Consumer Group
Consumer
Consumer
Consumer
Partition 4
Partition 5
Consumer
Consumer
Producer

questions
Kafka Message Loss
▪Kafka
-Failure >= then acknowledgements
-Use a higher ack level
▪Producer
-If disconnected from Kafka and buffer overflows
-Consider a producer side buffer (e.g. Flume)
▪Source
-If data retention is not enough before Producer consumes messages

Stream Processing
Considerations

questions
Processing
Storage Access
Custom
Producer
or
Processing &
Ingestion
Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series
Lookup
Batch
Processing
SQL
NRT REST
NRT
Dashboard

questions
Streaming agenda
▪ What do we mean by streaming?
▪ Streaming use-cases
▪ Streaming semantics
▪ Which streaming engine to choose?
▪ Streaming in our use-case

questions
What do we mean by streaming?
Constant low
milliseconds & under
Low milliseconds to
seconds, delay in
case of failures
10s of seconds or
more, re-run in case
of failures
Real-time Near real-time Batch

questions
But, there’s no free lunch
Constant low
milliseconds & under
Low milliseconds to
seconds, delay in
case of failures
10s of seconds or
more, re-run in case
of failures
Real-time Near real-time Batch
“Difficult” architectures, lower
latency
“Easier” architectures, higher
latency

questions
Streaming Use-cases
▪ Ingestion (most relevant in our use-case)
▪ Simple transformations
- Decision (e.g. anomaly detection)
- Enrichment (e.g. add a state based on zipcode)
▪ Advanced usage
- Machine Learning
- Windowing

questions
#1 - Simple ingestion
Buffer
Event e Stream
Processing Long term
storage
Event e

questions
#2 - Enrichment
Buffer
Event e Stream
Processing Storage
Event e’
e’ = enriched event e
Context store

questions
#2 - Decision
Buffer
Event e Stream
Processing Storage
Event e’
e’ = e + decision
Rules

questions
#3 – Advanced usage
Buffer
Event e Stream
Processing Storage
Event e’
e’ = aggregation or
windowed aggregation
Model

questions
#1 – Simple Ingestion
1. Zero transformation
- No transformation, plain ingest
- Keep the original format – SequenceFile, Text, etc.
- Allows to store data that may have errors in the schema
2. Format transformation
- Simply change the format of the field
- To a structured format, say, Avro, for example
- Can do schema validation
3. Atomic transformation
- Mask a credit card number

questions
#2 - Enrichment
Buffer
Event e Stream
Processing Storage
Event e’
e’ = enriched event e
Context store
Need to store
the context
somewhere

questions
Where to store the context?
1. Locally Broadcast Cached Dim Data
- Local to Process (On Heap, Off Heap)
- Local to Node (Off Process)
2. Partitioned Cache
- Shuffle to move new data to partitioned cache
3. External Fetch Data (e.g. HBase, Memcached)

questions
#1a - Locally broadcast cached data
Could be
On heap or Off
heap

questions
#1b - Off process cached data
Data is cached on
the node, outside of
process. Potentially
in an external
system like Rocks
DB

questions
#2 - Partitioned cache data
Data is partitioned
based on field(s)
and then cached

questions
#3 - External fetch
Data fetched from
external system

questions
Partitioned cache + external

questions
Delivery Types
▪ At most once
- Not good for many cases
- Only where performance/SLA is more important than accuracy
▪ Exactly once
- Expensive to achieve but desirable
▪ At least once
- Easiest to achieve

questions
Semantics of our architecture
Source System 1
Destination
systemSource System 2
Source System 3
Ingest Extract Streaming
engine
Push
Message broker

questions
Classification of storage systems
▪ File based
- S3
- HDFS
▪ NoSQL
- HBase
- Cassandra
▪ Document based
- Search
▪ NoSQL-SQL
- Kudu

questions
Classification of storage systems
▪ File based
- S3
- HDFS
▪ NoSQL
- HBase
- Cassandra
▪ Document based
- Search
▪ NoSQL-SQL
- Kudu
De-duplication at file level
Semantics at key/record level

Which streaming
engine to choose?

questions
Processing
Storage Access
Custom
Producer
or
Processing &
Ingestion
Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series
Lookup
Batch
Processing
SQL
NRT REST
NRT
Dashboard
Apache
Beam
Kafka
Streams

questions
Spark Streaming
▪ Micro batch based architecture
▪ Allows stateful transformations
▪ Feature rich
- Windowing
- Sessionization
- ML
- SQL (Structured Streaming)

questions
Spark Streaming
DStream
DStream
DStream
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count Print
Source Receiver RDD
RDD
RDD
Single Pass
Filter Count Print
First
Batc
h
Secon
d
Batch

questions
DStream
DStream
DStream
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count
Print
Source Receiver
RDD
partitions
RDD
Parition
RDD
Single Pass
Filter Count
Pre-first
Batch
First
Batc
h
Secon
d
Batch
Stateful
RDD 1
Print
Stateful
RDD 2
Stateful
RDD 1
Spark Streaming

questions
Flink
▪ True “streaming” system, but not as feature rich as Spark
▪ Much better event time handling
▪ Good built-in backpressure support
▪ Allows stateful transformations
▪ Lower Latency
- No Micro Batching
- Asynchronous Barrier Snapshotting (ABS)

questions
Flink - ABS
Operator
Buffer

questions
Operator
Buffer
Operator
Buffer
Flink - ABS
Barrier 1A
Hit
Barrier 1B
Still Behind

questions
Operator
Buffer
Flink - ABS
Both Barriers
Hit
Operator
Buffer
Barrier 1A
Hit
Barrier 1B
Still Behind

questions
Operator
Buffer
Flink - ABS Both Barriers
Hit
Operator
Buffer Barrier is
combined
and can
move on
Buffer can
be flushed
out

questions
Storm
▪ Old school
▪ Didn’t manage state – had to use Trident
▪ No good support for batch processing

questions
Samza
▪ Good integration with Kafka
▪ Doesn’t support batch
▪ Forked by Kafka Streams

questions
Flume
▪ Well integrated with the Hadoop ecosystem
▪ Allowed interceptors (for simple transformations)
▪ Supports buffering
- Memory
- File
- Kafka
▪ But no real fault-tolerance
▪ No state management

questions
Others
▪ Apache Apex
▪ Kafka Streams
▪ Heron

questions
Apache Beam
▪ Abstraction on top of Streaming Engines
▪ Best support for Google Dataflow

questions
Spark Streaming
▪ We chose Spark Streaming because:
- Same execution engine for batch and streaming
- Similar code for batch and streaming
- Support for security, kafka integration
- Thriving community
- We don’t have low millisecond requirements

questions
Processing
Storage Access
Custom
Producer
or
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series
Lookup
Batch
Processing
SQL
NRT REST
NRT
Dashboard

questions
Structured Landing Zones
Hive Relational Model
Kudu/HDFS
Hive Nested Model
HDFS
Aggregations
Kudu
HBase Entity Time
Series
Solr
Traditional SQL
Optimized for nested Structures like JSON
Optimized Storing and mutating aggregates
Optimized Entity 360 and time base access
Optimized faceted charts and reverse index
look ups

questions
Relational
▪ Everyone knows it
▪ Simple
▪ Very painful to do large Join
▪ May lead to customers making bad queries
▪ Easier to mutate

questions
Kudu Data Models
▪ Entity Summary Tables
- Quick update and access of aggregate of Entity Stats
▪ Event Tables
- Number of Partitioning strategies
- Partition by Entity
- Partition by Hash on time

questions
Kudu: Table Creation Example
CREATE EXTERNALTABLEny_taxi_trip(
vender_id STRING,
tpep_pickup_datetime TIMESTAMP,
tpep_dropoff_datetimeTIMESTAMP,
passenger_count INT,
trip_distanceDOUBLE,
pickup_longitudeDOUBLE,
pickup_latitudeDOUBLE,
rate_code_idSTRING,
store_and_fwd_flag STRING,
dropoff_longitudeDOUBLE,
dropoff_latitudeDOUBLE,
payment_typeSTRING,
fare_amount DOUBLE,
extra DOUBLE,
mta_tax DOUBLE,
improvement_surcharge DOUBLE,
tip_amountDOUBLE,
tolls_amountDOUBLE,
total_amountDOUBLE
)
STORED AS PARQUET
LOCATION 'usr/root/hive/ny_taxi_trip';

questions
Kudu: Data Population
SparkStreamingTaxiTripToKudu.scala

questions
Kudu: REST API
KuduServiceLayer.scala

questions
View Strategies
Hive Relational Model
Hive Nested Model
Models
Hive Normal Views
Hive Materialized
Table Views
Use in the cases where the view
requires a join that is done through a
shuffle
Use only for tables that filter
records/columns or use for marking
fields

questions
Nested
▪ Less Space than Denormalization
▪ Still have tables but the cost of joins is all but gone
▪ Also great for cartesian joins
- N x M vs N + M
▪ Not really supported yet with Kudu or HBase with SQL

questions
Nested Writing Example in Spark
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate"},
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "5001","type": "None"},
{ "id": "5002","type": "Glazed"},
{ "id": "5005","type": "Sugar"},
{ "id": "5007","type": "PowderedSugar"},
{ "id": "5006","type": "Chocolatewith Sprinkles"}
]

questions
Nested Writing Example in Spark
val jsonDF = hiveContext.read.json(jsonRDD)
jsonDF.write.parquet("./parquet")
hiveContext.createExternalTable("jsonNestedTable", "./parquet")

questions
Nested: Taxi Example
KuduToNestedHDFS.scala

questions
Entity Centric Time Series
▪ Partition by Entity ID
▪ Order by Time
▪ Allows for free windowing
▪ Allows for fetching of single time window of single entity at web scale

questions
HBase Entity Time Series
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40

questions
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Rest Call Short Scan

questions
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Mapper Mapper Mapper

questions
Cust-A, 10
Cust-A, 20
Cust-A, 40
Cust-C, 10
Cust-C, 20
Cust-C, 30
Cust-C, 40
Cust-B, 10
Cust-B, 20
Cust-B, 30
Cust-B, 40
Cust-F, 20
Cust-F, 30
Cust-F, 40
Cust-D, 10
Cust-D, 20
Cust-D, 40
Cust-G, 10
Cust-G, 20
Cust-G, 30
Cust-G, 40
Mapper
Mapper Mapper

questions
HBase: Row Key Example
def generateRowKey(customerTrans: CustomerTran, numOfSalts:Int): Array[Byte] = {
val salt = StringUtils.leftPad(
Math.abs(customerTrans.customerId.hashCode % numOfSalts).toString, 4, "0")
Bytes.toBytes(salt + ":" +
customerTrans.customerId + ":" +
StringUtils.leftPad(customerTrans.eventTimeStamp.toString, 11, "0") + ":trans:" +
customerTrans.transId)
}

questions
HBase: Population Example
SparkStreamingTaxiTripToHBase.scala

questions
HBase: REST Example
HBaseServiceLayer.scala

questions
Solr: Data Model
▪ Think of it like a cube on a object type
- In our case a taxi trip
- Allows for rollups and aggregations from object’s point of view
- Think of objects as immutable
- Try to find time based events
- May design more than one object type

questions
Solr Details
1 Trip:101 1
2 Trip:102 1
3 Trip:103 1
ID Document Live
4 Trip:104 1
5 Trip:105 1
ID Field Value Documents
1 Cash 1,3
2 Credit 2
3 Debit 4,5

questions
Single Value Aggregations
▪Get Array Lengths
ID Field Value Documents
1 Cash 1,3
2 Credit 2
3 Debit 4,5

questions
Multi Value Aggregations
▪Ordered Merge Join
-Think like a zipper
-Scans
-No Lookups
▪Top N from both sides
-Leaving the rest to other
▪Indexes distributed
▪No need to read document data
1 4 5 7 8 9 10 14 16
2 3 6 11 12 13 15 17 18
1 2 3 6 7 8 10 15 18
Cash
Credit
Vender A
4 5 9 11 12 13 14 16 17Vender B

questions
Solr: Population Example
SparkStreamingTaxiTripToSolR.scala

questions
Storage
Processing
Access
Custom
Producer
or

Batch Processing
Considerations

questions
Why have batch processing?
▪ When you need a larger context
- Say, to train a model
▪ Complex periodic job that does something
- Convert data to a nested structure for reduced number of shuffles
▪ In our use-case,
- Kudu -> HDFS Nested is batch processing
- KMeans calculation is also in bash

questions
Batch processing options
▪ Spark (+ MLlib)
▪ MapReduce (+ Mahout)
▪ Flink (+ Flink ML)

questions
Spark
▪ Pretty popular
▪ Much faster than MapReduce
▪ Thriving community

questions
MapReduce
▪ Sloooooow

questions
Flink
▪ Pretty popular
▪ Batch is a special case of Streaming
▪ Developing community

questions
In our use-case
▪ We chose Spark
- We were using Spark Streaming anyways
- Similar code between Spark and Spark Streaming
- Thriving community

Interactive
Data Access
Considerations

questions
Types of data access
▪ REST server/APIs for querying entities and aggregates
▪ UI for displaying search facets
▪ SQL engine

questions
Why have REST server?
▪ Tired of business people telling us how to access data
▪ Serves as an interface between the data engineers and business folks
▪ Lets business folks decide access patterns
▪ Engineers to optimize those patterns
▪ Brownie points from your boss
▪ And, it’s not that difficult to write!

questions
Don’t believe me?
import org.mortbay.jetty.Server
import org.mortbay.jetty.servlet.{Context, ServletHolder}
…
val server = new Server(port)
val sh = new ServletHolder(classOf[ServletContainer])
sh.setInitParameter("com.sun.jersey.config.property.resourceConfigClass",
"com.sun.jersey.api.core.PackagesResourceConfig")
sh.setInitParameter("com.sun.jersey.config.property.packages",
"com.hadooparchitecturebook.taxi360.server.hbase")
sh.setInitParameter("com.sun.jersey.api.json.POJOMappingFeature", "true”)
val context = new Context(server, "/", Context.SESSIONS)
context.addServlet(sh, "/*”)
server.start()
server.join()

questions
Then, write a ServiceLayer
@GET
@Path("vender/{venderId}/timeline")
@Produces(Array(MediaType.APPLICATION_JSON))
def getTripTimeLine (@PathParam("venderId") venderId:String,
@QueryParam("startTime") startTime:String = Long.MinValue.toString,
@QueryParam("endTime") endTime:String = Long.MaxValue.toString):
Array[NyTaxiYellowTrip] = {

questions
Use REST! Say no to business people!
▪ Access data like so:
http://<serverURL>:8080/vender/{venderId}/timeline

questions
UI requirements
Something that can
▪ Represent search results really well
▪ Integrates with Apache Solr on Hadoop

questions
UI options
▪ Hue
▪ Banana
▪ Kibana

questions
We choose Hue
▪ Because it’s included
▪ Please look at the others

questions
SQL engine criteria
▪ Low latency SQL access
▪ Allows for high concurrency
▪ JDBC/ODBC integration
▪ Capable of large scale aggregation
▪ Optionally integrates with Kudu for real-time updates to SQL tables

questions
Apache Hive
▪ Good JDBC integration
▪ Not really low latency, even when using Tez
▪ Doesn’t integrate with Kudu
▪ Can run at MR, Spark, or Tez

questions
Presto
▪ Low latency SQL engine from Facebook
▪ Provides JDBC/ODBC access
▪ Is only in-memory, large aggregations can lead to OOM errors

questions
Apache Impala
▪ Low latency SQL access
▪ Excellent concurrency support
▪ Integrates with Kudu for real-time SQL

questions
Apache Drill
▪ Similar in architecture to Impala

questions
Spark SQL
▪ Builds on top of Spark
▪ JDBC/ODBC access only via Spark Thrift Server
- Doesn’t scale well with larger number of concurrent users
- Doesn’t fully provide secure access.

questions
We choose
▪ Spark SQL
▪ Impala

questions
Processing
Storage Access
Custom
Producer
Processing &
Ingestion
Engine
Nested
Tables
Indexed
Cube
Relational
Tables
Entity Time
Series
Lookup
Batch
Processing
SQL
NRT Rest
NRT
Dashboard

questions
Storage
Processing
Custom
Producer
or
Access
Batch
Processing
SQL
NRT REST
NRT
Dashboard

questions
Access
Processing
Storage
Custom
Producer
or

questions
Processing
Storage Access
Custom
Producer
or

questions
High Level of the Demo Design
Producer
Kafka
Topic Foo
Partition 1
Partition 2
Partition 3
Spark
Streaming
Kudu
Spark
Streaming
HBase
Spark
Streaming
Solr
Spark
Streaming
HDFS
Kudu
HBase
Solr
HDFS
SQL
REST
REST
Hue
SQL

questions
Book Signing!
▪ 6:50 pm on Wednesday (tomorrow) at Cloudera booth

questions
Other Sessions
▪ Ask Us Anything session (all) – Thursday, 1:15 pm
▪ Top Five Mistakes When Writing Spark Applications – Wednesday, 1:15 pm

Thank you!
@hadooparchbook
tiny.cloudera.com/app-arch-slides
Jonathan Seidman | @jseidman
Ted Malaska | @ted_malaska
Mark Grover | @mark_grover

Hadoop application architectures - using Customer 360 as an example

More Related Content

What's hot

Viewers also liked

Similar to Hadoop application architectures - using Customer 360 as an example

Recently uploaded

Hadoop application architectures - using Customer 360 as an example