Apache Spark & Streaming

Apache Spark
Buenos Aires High Scalability
Buenos Aires, Argentina, Dic 2014
Fernando Rodriguez Olivera
@frodriguez

Professor at Universidad Austral (Distributed Systems, Compiler
Design, Operating Systems, …)
Creator of mvnrepository.com
Organizer at Buenos Aires High Scalability Group, Professor at
nosqlessentials.com
Twitter: @frodriguez

Apache Spark
Apache Spark is a Fast and General Engine
for Large-Scale data processing
In-Memory computing primitives
Supports for Batch, Interactive, Iterative and
Stream processing with Unified API

Apache Spark
Unified API for multiple kind of processing
Batch (high throughput)
Interactive (low latency)
Stream (continuous processing)
Iterative (results used immediately)

Daytona Gray Sort 100TB Benchmark
Data Size Time Nodes Cores
Hadoop MR
(2013)
102.5 TB 72 min 2,100
50,400
physical
Apache
Spark
(2014)
100 TB 23 min 206
6,592
virtualized
source: http://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html

Daytona Gray Sort 100TB Benchmark
Data Size Time Nodes Cores
Hadoop MR
(2013)
102.5 TB 72 min 2,100
50,400
physical
Apache
Spark
(2014)
100 TB 23 min 206
6,592
virtualized
3X faster using 10X fewer machines
source: http://databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html

Hadoop vs Spark for Iterative Proc
Logistic regression in Hadoop and Spark
source: https://spark.apache.org/

Hadoop MR Limits
Job Job Job
Hadoop HDFS
MapReduce designed for Batch Processing:
- Communication between jobs through FS
- Fault-Tolerance (between jobs) by Persistence to FS
- Memory not managed (relies on OS caches)
Compensated with: Storm, Samza, Giraph, Impala, Presto, etc

Apache Spark
Apache Spark (Core)
Spark
SQL
Spark
Streaming ML lib GraphX
Powered by Scala and Akka
APIs for Java, Scala, Python

Resilient Distributed Datasets (RDD)
RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
Immutable Collection of Objects

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
Partitioned and Distributed

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
Stored in Memory

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
Stored in Memory
Partitions Recomputed on Failure

RDD Transformations and Actions
RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
Compute
Function
(transformation)
e.g: apply
function
to count
chars

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
RDD of Ints
11
...
...
10
...
5
...
7
...
Compute
Function
(transformation)
e.g: apply
function
to count
chars

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
RDD of Ints
11
...
...
10
...
5
...
7
...
depends on
Compute
Function
(transformation)
e.g: apply
function
to count
chars

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
RDD of Ints
11
...
...
10
...
5
...
7
...
depends on
Compute
Function
(transformation)
e.g: apply
function
to count
chars
Int
N
Action

RDD of Strings
Hello World
...
...
A New Line
...
...
hello
The End
...
RDD of Ints
11
...
...
10
...
5
...
7
...
Compute
Function
(transformation)
e.g: apply
function
to count
chars
RDD Implementation
Partitions
Compute Function
Dependencies
Preferred Compute
Location
(for each partition)
Partitioner
depends on
Int
N
Action

Spark API
val spark = new SparkContext()
val lines = spark.textFile(“hdfs://docs/”) // RDD[String]
val nonEmpty = lines.filter(l => l.nonEmpty()) // RDD[String]
val count = nonEmpty.count
Scala
SparkContext spark = new SparkContext();
JavaRDD<String> lines = spark.textFile(“hdfs://docs/”)
JavaRDD<String> nonEmpty = lines.filter(l -> l.length() > 0);
long count = nonEmpty.count();
Java 8 Python
spark = SparkContext()
lines = spark.textFile(“hdfs://docs/”)
nonEmpty = lines.filter(lambda line: len(line) > 0)
count = nonEmpty.count()

RDD Operations
Transformations Actions
map(func)
flatMap(func)
filter(func)
take(N)
count()
collect()
groupByKey()
reduceByKey(func)
reduce(func)
mapValues(func)
takeOrdered(N)
top(N)
… …

Text Processing Example
Top Words by Frequency
(Step by step)

Create RDD from External Data
Apache Spark
Hadoop FileSystem,
I/O Formats, Codecs
HDFS S3 HBase MongoDB
Cassandra
…
Spark can read/write from any data source supported by Hadoop
I/O via Hadoop is optional (e.g: Cassandra connector bypass Hadoop)
// Step 1 - Create RDD from Hadoop Text File
val docs = spark.textFile(“/docs/”)
ElasticSearch

Function map
RDD[String] RDD[String]
Hello World
A New Line
hello
...
The end
.map(line => line.toLowerCase)
hello world
a new line
hello
...
the end
=
.map(_.toLowerCase)
// Step 2 - Convert lines to lower case
val lower = docs.map(line => line.toLowerCase)

Functions map and flatMap
RDD[String]
hello world
a new line
hello
...
the end

RDD[String]
hello world
a new line
hello
...
the end
.map( … )
RDD[Array[String]]
_.split(“s+”)
hello
a
hello
...
the
world
new line
end

RDD[String]
hello world
a new line
hello
...
the end
.map( … )
RDD[Array[String]]
_.split(“s+”)
hello
a
hello
...
the
world
new line
end
.flatten
RDD[String]
hello
world
a
new
line
...
*

hello world
a new line
hello
...
the end
RDD[Array[String]]
hello
.flatMap(line => line.split(“s+“))
RDD[String]
.map( … )
_.split(“s+”)
a
hello
...
the
world
new line
end
.flatten
RDD[String]
hello
world
a
new
line
...
*

RDD[String]
hello world
a new line
hello
...
the end
.map( … )
RDD[Array[String]]
_.split(“s+”)
hello
a
world
new line
hello
...
the
end
.flatten
.flatMap(line => line.split(“s+“))
RDD[String]
world
// Step 3 - Split lines into words
val words = lower.flatMap(line => line.split(“s+“))
Note: flatten() not available in spark, only flatMap
hello
a
new
line
...
*

Key-Value Pairs
RDD[Tuple2[String, Int]]
RDD[String] RDD[(String, Int)]
hello
world
a
new
line
hello
...
hello
world
a
new
line
hello
...
.map(word => Tuple2(word, 1))
1
1
1
1
1
1
=
.map(word => (word, 1))
// Step 4 - Split lines into words
val counts = words.map(word => (word, 1))
Pair RDD

Shuffling
RDD[(String, Int)]
hello
world
a
new
line
hello
1
1
1
1
1
1

Shuffling
hello
world
a
new
line
hello
1
1
1
1
1
1
RDD[(String, Iterator[Int])]
world
a
1
1
new 1
line
hello
1
1
.groupByKey
1
RDD[(String, Int)]

Shuffling
hello
world
a
new
line
hello
1
1
1
1
1
1
world
a
1
1
new 1
line
hello
1
1
.groupByKey
1
RDD[(String, Int)]
RDD[(String, Int)]
world
a
1
1
new 1
line
hello
1
2
.mapValues
_.reduce(…)
(a,b) => a+b

Shuffling
hello
world
a
new
line
hello
1
1
1
1
1
1
world
a
1
1
new 1
line
hello
1
1
.groupByKey
1
.reduceByKey((a, b) => a + b)
RDD[(String, Int)]
RDD[(String, Int)]
world
a
1
1
new 1
line
hello
1
2
.mapValues
_.reduce(…)
(a,b) => a+b

Shuffling
RDD[(String, Int)]
hello
world
a
new
line
hello
1
1
1
1
1
1
world
a
1
1
new 1
line
hello
1
1
.groupByKey
1
RDD[(String, Int)]
.reduceByKey((a, b) => a + b)
// Step 5 - Count all words
val freq = counts.reduceByKey(_ + _)
world
a
1
1
new 1
line
hello
1
2
.mapValues
_.reduce(…)
(a,b) => a+b

Top N (Prepare data)
RDD[(String, Int)] RDD[(Int, String)]
world
a
1
1
new 1
line
hello
1
2
.map(_.swap)
1
1
1 new
world
a
line
hello
1
2
// Step 6 - Swap tuples (partial code)
freq.map(_.swap)

Top N (First Attempt)
RDD[(Int, String)]
1
1
1 new
world
a
line
hello
1
2

RDD[(Int, String)]
1
1
1 new
world
a
line
hello
1
2
.sortByKey
RDD[(Int, String)]
2
1
1 a
hello
world
new
line
1
1
(sortByKey(false) for descending)

RDD[(Int, String)] Array[(Int, String)]
1
1
1 new
world
a
line
hello
1
2
hello
world
2
1
RDD[(Int, String)]
2
1
1 a
hello
world
.sortByKey .take(N)
new
line
1
1
(sortByKey(false) for descending)

Top N
Array[(Int, String)]
RDD[(Int, String)]
1
1
1 new
world
a
line
hello
1
2
world
a
1
1
.top(N)
hello
line
2
1
hello
line
2
1
local top N *
local top N *
reduction
* local top N implemented by bounded priority queues
// Step 6 - Swap tuples (complete code)
val top = freq.map(_.swap).top(N)

Top Words by Frequency (Full Code)
val spark = new SparkContext()
// RDD creation from external data source
val docs = spark.textFile(“hdfs://docs/”)
// Split lines into words
val lower = docs.map(line => line.toLowerCase)
val words = lower.flatMap(line => line.split(“s+“))
val counts = words.map(word => (word, 1))
// Count all words (automatic combination)
val freq = counts.reduceByKey(_ + _)
// Swap tuples and get top results
val top = freq.map(_.swap).top(N)
top.foreach(println)

RDD Persistence (in-memory)
RDD
…
...
...
…
...
…
...
…
...
.cache()
.persist()
.persist(storageLevel)
StorageLevel:
MEMORY_ONLY,
MEMORY_ONLY_SER,
MEMORY_AND_DISK,
MEMORY_AND_DISK_SER,
DISK_ONLY, …
(memory only)
(memory only)
(lazy persistence & caching)

RDD Lineage
RDD Transformations
words = sc.textFile(“hdfs://large/file/”) HadoopRDD
.map(_.toLowerCase)
.flatMap(_.split(“ “)) FlatMappedRDD
nums = words.filter(_.matches(“[0-9]+”))
alpha.count()
MappedRDD
alpha = words.filter(_.matches(“[a-z]+”))
FilteredRDD
FilteredRDD
Lineage
(built on the driver
by the transformations)
Action (run job on the cluster)

SchemaRDD & SQL
SchemaRDD
Row
...
...
Row
...
...
Row
Row
...
RRD of Row + Column Metadata
Queries with SQL
Support for Reflection, JSON,
Parquet, …

SchemaRDD & SQL
topWords
Row
...
...
Row
...
...
Row
Row
...
case class Word(text: String, n: Int)
val wordsFreq = freq.map {
case (text, count) => Word(text, count)
} // RDD[Word]
wordsFreq.registerTempTable("wordsFreq")
val topWords = sql("select text, n
from wordsFreq
order by n desc
limit 20”) // RDD[Row]
topWords.collect().foreach(println)

Spark Streaming
DStream
RDD RDD RDD RDD RDD RDD
Data Collected, Buffered and Replicated
by a Receiver (one per DStream)
then Pushed to a stream as small RDDs
Configurable Batch Intervals.
e.g: 1 second, 5 seconds, 5 minutes
Receiver
e.g: Kafka,
Kinesis,
Flume,
Sockets,
Akka
etc

DStream Transformations
DStream
DStream
transform
Receiver
// Example
val entries = stream.transform { rdd => rdd.map(Log.parse) }
// Alternative
val entries = stream.map(Log.parse)

Parallelism with Multiple Receivers
DStream 1
Receiver 1 RDD RDD RDD RDD RDD RDD
DStream 2
Receiver 2 RDD RDD RDD RDD RDD RDD
union of (stream1, stream2, …)
Union can be used to manage multiple DStreams as
a single logical stream

Sliding Windows
DStream
DStream
… … … W3 W2 W1
Window Length: 3, Sliding Interval: 1
Receiver

Deployment with Hadoop
A
B
C
D
/large/file
allocates resources
(cores and memory)
Spark
Worker
Data
Node 1
Application
Spark
Worker
Data
Node 3
Spark
Worker
Data
Node 4
Spark
Worker
Data
Node 2
A C B C A B A B
Spark
Master
Name
Node
RF 3 D D D C
Client
Submit App
(mode=cluster)
Driver Executors Executors Executors
DN + Spark
HDFS Spark

twitter: @frodriguez

Apache Spark & Streaming

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Apache Spark & Streaming (20)

Recently uploaded (20)

Apache Spark & Streaming