Abstract image of a woman sitting on books and studying on a laptop

Rethinking Streaming Analytics For Scale

Helena Edelson, profile picture
Helena Edelson

The document discusses the need for scalable and fault-tolerant data processing systems to manage large volumes of disparate data for applications like cybersecurity and big data analytics. It critiques traditional ETL processes and promotes a unified architecture that combines batch and stream processing to simplify analytics handling. Key strategies and technologies for implementing such systems, including Spark, Cassandra, and Kafka, are also highlighted.

Technology
Related topics:
Data Science Insights • Data Pipelines
1 of 100
Downloaded 256 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
RETHINKING STREAMING ANALYTICS FOR SCALE Helena Edelson 1 @helenaedelson
@helenaedelson Who Is This Person? • VP of Product Engineering @Tuplejump • Big Data, Analytics, Cloud Engineering, Cyber Security • Committer / Contributor to FiloDB, Spark Cassandra Connector, Akka, Spring Integration 2 • @helenaedelson • github.com/helena • linkedin.com/in/helenaedelson • slideshare.net/helenaedelson
@helenaedelson
@helenaedelson Tuplejump - Open Source github.com/tuplejump • FiloDB - part of this talk • Calliope - the first Spark-Cassandra integration • Stargate - an open source Lucene indexer for Cassandra • SnackFS - open source HDFS for Cassandra 4
@helenaedelson What Will We Talk About • The Problem Domain • Example Context • Rethinking Architecture • We don't have to look far to look back • Streaming & Data Science • Challenging Assumptions • Revisiting the goal and the stack • Integration • Simplification 5
@helenaedelson THE PROBLEM DOMAIN Delivering Meaning From A Flood Of Data 6 @helenaedelson
@helenaedelson The Problem Domain Need to build scalable, fault tolerant, distributed data processing systems that can handle massive amounts of data from disparate sources, with different data structures. 7
@helenaedelson Translation How to build adaptable, elegant systems for complex analytics and learning tasks to run as large-scale clustered dataflows 8
@helenaedelson How Much Data Yottabyte = quadrillion gigabytes or septillion bytes We all have a lot of data • Terabytes • Petabytes... http://en.wikipedia.org/wiki/Yottabyte 9 100 trillion $ in DC fees
@helenaedelson Delivering Meaning • Deliver meaning in sec/sub-sec latency • Disparate data sources & schemas • Billions of events per second • High-latency batch processing • Low-latency stream processing • Aggregation of historical from the stream 10
@helenaedelson While We Monitor, Predict & Proactively Handle • Massive event spikes & bursty traffic • Fast producers / slow consumers • Network partitioning & out of sync systems • DC down • Wait, we've DDOS'd ourselves from fast streams? • Autoscale issues – When we scale down VMs how do we not lose data? 11
@helenaedelson And stay within our AWS / Rackspace budget 12
@helenaedelson EXAMPLE CONTEXT: CYBER SECURITY Hunting The Hunter 13
@helenaedelson • Track activities of international threat actor groups, nation-state, criminal or hactivist • Intrusion attempts • Actual breaches • Profile adversary activity • Analysis to understand their motives, anticipate actions and prevent damage Adversary Profiling & Hunting: Online & Offline 14
@helenaedelson • Machine events • Endpoint intrusion detection • Anomalies/indicators of attack or compromise • Machine learning • Training models based on patterns from historical data • Predict potential threats • profiling for adversary Identification Stream Processing 15
@helenaedelson Data Requirements & Description • Streaming event data • Log messages • User activity records • System ops & metrics data • Disparate data sources • Wildly differing data structures 16
@helenaedelson Massive Amounts Of Data • One machine can generate 2+ TB per day • Tracking millions of devices • 1 million writes per second - bursty • High % writes, lower % reads 17
@helenaedelson RETHINKING ARCHITECTURE 18
@helenaedelson 19 few years in Silicon Valley Cloud Engineering team @helenaedelson
@helenaedelson 20 Batch analytics data flow from several years ago looked like...
@helenaedelson 21 Batch analytics data flow from several years ago looked like...
@helenaedelson 22 Transforming data multiple times, multiple ways
@helenaedelson 23 Sweet, let's triple the code we have to update and regression test every time our analytics logic changes
@helenaedelson STREAMING & DATA SCIENCE Enter Streaming for Big Data 24
@helenaedelson Streaming: Big Data, Fast Data, Fast Timeseries Data • Reactive processing of data as it comes in to derive instant insights • Is this enough? • Need to combine with existing big data, historical processing, ad hoc queries 25
@helenaedelson New Requirements, Common Use Case I need fast access to historical data on the fly for predictive modeling with real time data from the stream 26
@helenaedelson It's Not A Stream It's A Flood • Netflix • 50 - 100 billion events per day • 1 - 2 million events per second at peak • LinkedIn • 500 billion write events per day • 2.5 trillion read events per day • 4.5 million events per second at peak with Kafka • 1 PB of stream data 27
@helenaedelson Which Translates To • Do it fast • Do it cheap • Do it at scale 28
@helenaedelson Oh, and don't loose data 29
@helenaedelson AND THEN WE GREEKED OUT 30 Lambda
@helenaedelson Lambda Architecture A data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods. 31 • Or, "How to beat the CAP theorum" • An approach coined by Nathan Mars • This was a huge stride forward
@helenaedelson • Doing complex asynchronous transformations • That need to run with low latency (say, a few seconds to a few hours) • Examples • Weather analytics and prediction system • News recommendation system 32 Applications Using Lambda Architecture
@helenaedelson https://www.mapr.com/developercentral/lambda-architecture 33
@helenaedelson Implementing Is Hard • Real-time pipeline backed by KV store for updates • Many moving parts - KV store, real time, batch • Running similar code in two places • Still ingesting data to Parquet/HDFS • Reconcile queries against two different places 34
@helenaedelson Performance Tuning & Monitoring on so many disparate systems 35 Also Hard
@helenaedelson 36 λ: Streaming & Batch Flows Evolution Or Just Addition? Or Just Technical Debt?
@helenaedelson Lambda Architecture Ingest an immutable sequence of records is captured and fed into • a batch system • and a stream processing system in parallel 37
@helenaedelson WAIT, DUAL SYSTEMS? 38 Challenge Assumptions
@helenaedelson Which Translates To • Performing analytical computations & queries in dual systems • Duplicate Code • Untyped Code - Strings • Spaghetti Architecture for Data Flows • One Busy Network 39
@helenaedelson 40 Why? • Why support code, machines and running services of two analytics systems? • Is a separate batch system needed? • Can we do everything in a streaming system?
@helenaedelson YES 41 • A unified system for streaming and batch • Real-time processing and reprocessing • Code changes • Fault tolerance http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html - Jay Kreps
@helenaedelson ANOTHER ASSUMPTION: ETL 42 Challenge Assumptions
@helenaedelson Extract, Transform, Load (ETL) • Extraction of data from one system into another • Transforming it • Loading it into another system 43
@helenaedelson ETL • Each step can introduce errors and risk • Writing intermediary files • Parsing and re-parsing plain text • Tools can cost millions of dollars • Decreases throughput • Increased complexity • Can duplicate data after failover 44
@helenaedelson Extract, Transform, Load (ETL) "Designing and maintaining the ETL process is often considered one of the most difficult and resource- intensive portions of a data warehouse project." http://docs.oracle.com/cd/B19306_01/server.102/b14223/ettover.htm 45 Also unnecessarily redundant and often typeless
@helenaedelson And let's duplicate the pattern over all our DataCenters 46
@helenaedelson 47 These are not the solutions you're looking for
@helenaedelson REVISITING THE GOAL 48
@helenaedelson Removing The 'E' in ETL Thanks to technologies like Avro and Protobuf we don’t need the “E” in ETL. Instead of text dumps that you need to parse over multiple systems: E.g Scala and Avro • A return to strong typing in the big data ecosystem • Can work with binary data that remains strongly typed 49
@helenaedelson Removing The 'L' in ETL If data collection is backed by a distributed messaging system (e.g. Kafka) you can do real-time fanout of the ingested data to all consumers. No need to batch "load". • From there each consumer can do their own transformations 50
@helenaedelson 51 #NoMoreGreekLetterArchitectures
@helenaedelson NoETL 52
@helenaedelson Pick Technologies Wisely Based on your requirements • Latency • Real time / Sub-Second: < 100ms • Near real time (low): > 100 ms or a few seconds - a few hours • Consistency • Highly Scalable • Topology-Aware & Multi-Datacenter support • Partitioning Collaboration - do they play together well 53
@helenaedelson And Remember • Flows erode • Entropy happens • "Everything fails, all the time" - Kyle Kingsbury 54
@helenaedelson REVISITING THE STACK 55
@helenaedelson Stream Processing & Frameworks 56 + GearPump
@helenaedelson Strategies 57 • Partition For Scale & Data Locality • Replicate For Resiliency • Share Nothing • Fault Tolerance • Asynchrony • Async Message Passing • Memory Management • Data lineage and reprocessing in runtime • Parallelism • Elastically Scale • Isolation • Location Transparency
@helenaedelson Fault Tolerance • Graceful service degradation • Data integrity / accuracy under failure • Resiliency during traffic spikes • Pipeline congestion / bottlenecks • Easy to debug and find failure source • Easy to deploy 58
@helenaedelson Strategy Technologies Scalable Infrastructure / Elastic Spark, Cassandra, Kafka Partition For Scale, Network Topology Aware Cassandra, Spark, Kafka, Akka Cluster Replicate For Resiliency Spark,Cassandra, Akka Cluster all hash the node ring Share Nothing, Masterless Cassandra, Akka Cluster both Dynamo style Fault Tolerance / No Single Point of Failure Spark, Cassandra, Kafka Replay From Any Point Of Failure Spark, Cassandra, Kafka, Akka + Akka Persistence Failure Detection Cassandra, Spark, Akka, Kafka Consensus & Gossip Cassandra & Akka Cluster Parallelism Spark, Cassandra, Kafka, Akka Asynchronous Data Passing Kafka, Akka, Spark Fast, Low Latency, Data Locality Cassandra, Spark, Kafka Location Transparency Akka, Spark, Cassandra, Kafka My Nerdy Chart 59
@helenaedelson SMACK • Scala & Spark Streaming • Mesos • Akka • Cassandra • Kafka 60
@helenaedelson Spark Streaming • One runtime for streaming and batch processing • Join streaming and static data sets • No code duplication • Easy, flexible data ingestion from disparate sources to disparate sinks • Easy to reconcile queries against multiple sources • Easy integration of KV durable storage 61
@helenaedelson Training Data Feature Extraction Model Training Model Testing Test Data Your Data Extract Data To Analyze Train your model to predict 62 val context = new StreamingContext(conf, Milliseconds(500)) val model = KMeans.train(dataset, ...) // learn offline val stream = KafkaUtils .createStream(ssc, zkQuorum, group,..) .map(event => model.predict(event.feature))
@helenaedelson 63 High performance concurrency framework for Scala and Java • Fault Tolerance • Asynchronous messaging and data processing • Parallelization • Location Transparency • Local / Remote Routing • Akka: Cluster / Persistence / Streams
@helenaedelson Akka Actors 64 A distribution and concurrency abstraction • Compute Isolation • Behavioral Context Switching • No Exposed Internal State • Event-based messaging • Easy parallelism • Configurable fault tolerance
@helenaedelson High Performance Streaming Built On Akka • Apache Flink - uses Akka for • Actor model and hierarchy, Deathwatch and distributed communication between job and task managers • GearPump - models the entire streaming system with an actor hierarchy • Supervision, Isolation, Concurrency 65
@helenaedelson Apache Cassandra • Extremely Fast • Extremely Scalable • Multi-Region / Multi-Datacenter • Always On • No single point of failure • Survive regional outages • Easy to operate • Automatic & configurable replication 66
@helenaedelson 90% of streaming data at Netflix is stored in Cassandra 67
@helenaedelson STREAM INTEGRATION 68
@helenaedelson 69 KillrWeather http://github.com/killrweather/killrweather A reference application showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments. http://github.com/databricks/reference-apps/tree/master/timeseries/scala/timeseries-weather/src/main/scala/com/ databricks/apps/weather
@helenaedelson val context = new StreamingContext(conf, Seconds(1)) val stream = KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder]( context, kafkaParams, kafkaTopics) stream.flatMap(func1).saveToCassandra(ks1,table1) stream.map(func2).saveToCassandra(ks1,table1) context.start() 70 Kafka, Spark Streaming and Cassandra
@helenaedelson class KafkaProducerActor[K, V](config: ProducerConfig) extends Actor {
 
 override val supervisorStrategy =
 OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) {
 case _: ActorInitializationException => Stop
 case _: FailedToSendMessageException => Restart case _: ProducerClosedException => Restart case _: NoBrokersForPartitionException => Escalate case _: KafkaException => Escalate
 case _: Exception => Escalate
 } 
 private val producer = new KafkaProducer[K, V](producerConfig)
 
 override def postStop(): Unit = producer.close() 
 def receive = {
 case e: KafkaMessageEnvelope[K,V] => producer.send(e)
 }
 } 71 Kafka, Spark Streaming, Cassandra & Akka
@helenaedelson Spark Streaming, ML, Kafka & C* val ssc = new StreamingContext(new SparkConf()…, Seconds(5)
 val testData = ssc.cassandraTable[String](keyspace,table).map(LabeledPoint.parse)
 
 val trainingStream = KafkaUtils.createStream[K, V, KDecoder, VDecoder](
 ssc, kafkaParams, topicMap, StorageLevel.MEMORY_ONLY) .map(_._2).map(LabeledPoint.parse) trainingStream.saveToCassandra("ml_keyspace", "raw_training_data")
 
 val model = new StreamingLinearRegressionWithSGD()
 .setInitialWeights(Vectors.dense(weights))
 .trainOn(trainingStream) //Making predictions on testData model .predictOnValues(testData.map(lp => (lp.label, lp.features))) .saveToCassandra("ml_keyspace", "predictions") 72
@helenaedelson STREAM INTEGRATION: DATA LOCALITY & TIMESERIES 73 SMACK
@helenaedelson 74
@helenaedelson 75 class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 } Kafka, Spark Streaming, Cassandra & Akka
@helenaedelson class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 } 76 Now we can replay • On failure • Reprocessing on code changes • Future computation...
@helenaedelson 77 Here we are pre-aggregating to a table for fast querying later - in other secondary stream aggregation computations and scheduled computing class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 }
@helenaedelson CREATE TABLE weather.raw_data (
 wsid text, year int, month int, day int, hour int, 
 temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, one_hour_precip PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); CREATE TABLE daily_aggregate_precip (
 wsid text,
 year int,
 month int,
 day int,
 precipitation counter,
 PRIMARY KEY ((wsid), year, month, day)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC); Data Model (simplified) 78
@helenaedelson 79 class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 } Gets the partition key: Data Locality Spark C* Connector feeds this to Spark Cassandra Counter column in our schema, no expensive `reduceByKey` needed. Simply let C* do it: not expensive and fast.
@helenaedelson The Thing About S3 "Amazon S3 is a simple key-value store" - docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html • Keys 2015/05/01 and 2015/05/02 do not live in the “same place” • You can roll your own with AmazonS3Client and do the heavy lifting yourself and throw that data into Spark 80
@helenaedelson CREATE TABLE weather.raw_data (
 wsid text, year int, month int, day int, hour int, 
 temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, one_hour_precip PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); C* Clustering Columns Writes by most recent Reads return most recent first Timeseries Data 81 Cassandra will automatically sort by most recent for both write and read
@helenaedelson 82 val multipleStreams = for (i <- numstreams) { streamingContext.receiverStream[HttpRequest](new HttpReceiver(port)) } streamingContext.union(multipleStreams) .map { httpRequest => TimelineRequestEvent(httpRequest)} .saveToCassandra("requests_ks", "timeline") CREATE TABLE IF NOT EXISTS requests_ks.timeline ( timesegment bigint, url text, t_uuid timeuuid, method text, headers map <text, text>, body text, PRIMARY KEY ((url, timesegment) , t_uuid) ); Record Every Event In The Order In Which It Happened, Per URL timesegment protects from writing unbounded partitions. timeuuid protects from simultaneous events over-writing one another.
@helenaedelson val stream = KafkaUtils.createDirectStream(...)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw) stream .map(hour => (hour.id, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 83 Replay and Reprocess - Any Time Data is on the nodes doing the querying - Spark C* Connector - Partitions • Timeseries data with Data Locality • Co-located Spark + Cassandra nodes • S3 does not give you Cassandra & Spark Streaming: Data Locality For Free®
@helenaedelson class PrecipitationActor(ssc: StreamingContext, settings: Settings) extends AggregationActor {
 import akka.pattern.pipe 
 def receive : Actor.Receive = { 
 case GetTopKPrecipitation(wsid, year, k) => topK(wsid, year, k, sender)
 }
 
 /** Returns the 10 highest temps for any station in the `year`. */
 def topK(wsid: String, year: Int, k: Int, requester: ActorRef): Unit = {
 val toTopK = (aggregate: Seq[Double]) => TopKPrecipitation(wsid, year,
 ssc.sparkContext.parallelize(aggregate).top(k).toSeq)
 
 ssc.cassandraTable[Double](keyspace, dailytable)
 .select("precipitation")
 .where("wsid = ? AND year = ?", wsid, year)
 .collectAsync().map(toTopK) pipeTo requester
 }
 } 84 Queries pre-aggregated tables from the stream Compute Isolation: Actor
@helenaedelson 85 class TemperatureActor(sc: SparkContext, settings: Settings) extends AggregationActor {
 import akka.pattern.pipe 
 def receive: Actor.Receive = {
 case e: GetMonthlyHiLowTemperature => highLow(e, sender)
 }
 
 def highLow(e: GetMonthlyHiLowTemperature, requester: ActorRef): Unit =
 sc.cassandraTable[DailyTemperature](keyspace, daily_temperature_aggr)
 .where("wsid = ? AND year = ? AND month = ?", e.wsid, e.year, e.month)
 .collectAsync()
 .map(MonthlyTemperature(_, e.wsid, e.year, e.month)) pipeTo requester } C* data is automatically sorted by most recent - due to our data model. Additional Spark or collection sort not needed. Efficient Batch Analysis
@helenaedelson A NEW APPROACH 86 Simplification
@helenaedelson Everything On The Streaming Platform 87
@helenaedelson Reprocessing • Start a new stream job to re-process from the beginning • Save re-processed data as a version table • Application should then read from new version table • Stop old version of the job, and delete the old table 88
@helenaedelson One Pipeline For Fast & Big Data 89 • How do I make the SMACK stack work for ML, Ad-Hoc + Fast Data? • How do I combine Spark Streaming + Ad Hoc and have good performance?
@helenaedelson FiloDB Designed to ingest streaming data, including machine, event, and time-series data, and run very fast analytical queries over them. 90 • Distributed, versioned, columnar analytics database • Built for fast streaming analytics & OLAP • Currently based on Apache Cassandra & Spark • github.com/tuplejump/FiloDB
@helenaedelson Breakthrough Performance For Analytical Queries • Queries run in parallel in Spark for scale-out ad-hoc analysis • Fast for interactive data science and ad hoc queries • Up to 200x Faster Queries for Spark on Cassandra 2.x • Parquet Performance with Cassandra Flexibility • Increased performance ceiling coming 91
@helenaedelson Versioning & Why It Matters • Immutability • Databases: let's mutate one giant piece of state in place • Basically hasn't changed since 1970's! • With Big Data and streaming, incremental processing is increasingly important 92
@helenaedelson FiloDB Versioning FiloDB is built on functional principles and lets you version and layer changes • Incrementally add a column or a few rows as a new version • Add changes as new versions, don't mutate! • Writes are idempotent - exactly once ingestion • Easily control what versions to query • Roll back changes inexpensively • Stream out new versions as continuous queries :) 93
@helenaedelson No Cassandra? Keep All In Memory 94 • Unlike RDDs and DataFrames, FiloDB can ingest new data, and still be fast • Unlike RDDs, FiloDB can filter in multiple ways, no need for entire table scan
@helenaedelson Spark Streaming to FiloDB val ratingsStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder(
 ssc, kafkaParams, topics)
 
 ratingsStream.foreachRDD { (message: RDD[(String, String)], batchTime: Time) =>
 val df = message
 .map(_._2.split(","))
 .map(rating => Rating(trim(rating))
 .toDF("fromuserid", "touserid", "rating")
 
 // add the batch time to the DataFrame
 val dfWithBatchTime = df.withColumn(
 "batch_time", org.apache.spark.sql.functions.lit(batchTime.milliseconds))
 
 // save the DataFrame to FiloDB
 dfWithBatchTime.write.format("filodb.spark")
 .option("dataset", "ratings")
 .save()
 } 95 .dfWithBatchTime.write.format("org.apache.spark.sql.cassandra")
@helenaedelson Architectyr? "This is a giant mess" - Going Real-time - Data Collection and Stream Processing with Apache Kafka, Jay Kreps 96
@helenaedelson 97
@helenaedelson 98
@helenaedelson 99 @helenaedelson github.com/helena helena@tuplejump.com slideshare.net/helenaedelson THANKS!
@helenaedelson

More Related Content

PDF
Sa introduction to big data pipelining with cassandra &amp; spark west mins...
Simon Ambridge
 
PDF
Leveraging Kafka for Big Data in Real Time Bidding, Analytics, ML & Campaign ...
Helena Edelson
 
PDF
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Helena Edelson
 
PDF
Streaming Big Data & Analytics For Scale
Helena Edelson
 
PPTX
Intro to Apache Spark
Mammoth Data
 
PDF
NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch Analysis
Helena Edelson
 
PDF
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Natalino Busa
 
PDF
Tachyon and Apache Spark
rhatr
 
Sa introduction to big data pipelining with cassandra &amp; spark west mins...
Simon Ambridge
 
Leveraging Kafka for Big Data in Real Time Bidding, Analytics, ML & Campaign ...
Helena Edelson
 
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Helena Edelson
 
Streaming Big Data & Analytics For Scale
Helena Edelson
 
Intro to Apache Spark
Mammoth Data
 
NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch Analysis
Helena Edelson
 
Real-Time Anomaly Detection with Spark MLlib, Akka and Cassandra
Natalino Busa
 
Tachyon and Apache Spark
rhatr
 

What's hot (20)

PDF
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Lucidworks
 
PPTX
Re-envisioning the Lambda Architecture : Web Services & Real-time Analytics ...
Brian O'Neill
 
PDF
Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
Anton Kirillov
 
PPTX
Real Time Data Processing Using Spark Streaming
Hari Shreedharan
 
PDF
Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...
Helena Edelson
 
PPTX
Alpine academy apache spark series #1 introduction to cluster computing wit...
Holden Karau
 
PDF
Reactive app using actor model & apache spark
Rahul Kumar
 
PDF
Reactive dashboard’s using apache spark
Rahul Kumar
 
PPTX
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Robert "Chip" Senkbeil
 
PDF
Akka in Production - ScalaDays 2015
Evan Chan
 
PDF
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Helena Edelson
 
PPTX
Lambda architecture with Spark
Vincent GALOPIN
 
PPTX
Real time data viz with Spark Streaming, Kafka and D3.js
Ben Laird
 
PDF
Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...
DataStax Academy
 
PDF
Kafka spark cassandra webinar feb 16 2016
Hiromitsu Komatsu
 
PPTX
Lambda architecture on Spark, Kafka for real-time large scale ML
huguk
 
PDF
Apache Spark 1.6 with Zeppelin - Transformations and Actions on RDDs
Timothy Spann
 
PDF
Big Data visualization with Apache Spark and Zeppelin
prajods
 
PDF
Building Reactive Distributed Systems For Streaming Big Data, Analytics & Mac...
Helena Edelson
 
PDF
SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai
Codemotion Dubai
 
Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...
Lucidworks
 
Re-envisioning the Lambda Architecture : Web Services & Real-time Analytics ...
Brian O'Neill
 
Data processing platforms architectures with Spark, Mesos, Akka, Cassandra an...
Anton Kirillov
 
Real Time Data Processing Using Spark Streaming
Hari Shreedharan
 
Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...
Helena Edelson
 
Alpine academy apache spark series #1 introduction to cluster computing wit...
Holden Karau
 
Reactive app using actor model & apache spark
Rahul Kumar
 
Reactive dashboard’s using apache spark
Rahul Kumar
 
Spark Kernel Talk - Apache Spark Meetup San Francisco (July 2015)
Robert "Chip" Senkbeil
 
Akka in Production - ScalaDays 2015
Evan Chan
 
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Helena Edelson
 
Lambda architecture with Spark
Vincent GALOPIN
 
Real time data viz with Spark Streaming, Kafka and D3.js
Ben Laird
 
Typesafe & William Hill: Cassandra, Spark, and Kafka - The New Streaming Data...
DataStax Academy
 
Kafka spark cassandra webinar feb 16 2016
Hiromitsu Komatsu
 
Lambda architecture on Spark, Kafka for real-time large scale ML
huguk
 
Apache Spark 1.6 with Zeppelin - Transformations and Actions on RDDs
Timothy Spann
 
Big Data visualization with Apache Spark and Zeppelin
prajods
 
Building Reactive Distributed Systems For Streaming Big Data, Analytics & Mac...
Helena Edelson
 
SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai
Codemotion Dubai
 
Ad

Viewers also liked (9)

PDF
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Helena Edelson
 
PDF
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Helena Edelson
 
PDF
How to deploy Apache Spark 
to Mesos/DCOS
Legacy Typesafe (now Lightbend)
 
PPTX
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Spark Summit
 
PDF
Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization
Patrick Di Loreto
 
PDF
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Legacy Typesafe (now Lightbend)
 
PDF
H2O - the optimized HTTP server
Kazuho Oku
 
PDF
Container Orchestration Wars
Karl Isenberg
 
PDF
Linux 4.x Tracing Tools: Using BPF Superpowers
Brendan Gregg
 
Lambda Architecture with Spark Streaming, Kafka, Cassandra, Akka, Scala
Helena Edelson
 
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Helena Edelson
 
How to deploy Apache Spark 
to Mesos/DCOS
Legacy Typesafe (now Lightbend)
 
Data Science lifecycle with Apache Zeppelin and Spark by Moonsoo Lee
Spark Summit
 
Using Spark, Kafka, Cassandra and Akka on Mesos for Real-Time Personalization
Patrick Di Loreto
 
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
Legacy Typesafe (now Lightbend)
 
H2O - the optimized HTTP server
Kazuho Oku
 
Container Orchestration Wars
Karl Isenberg
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Brendan Gregg
 
Ad

Similar to Rethinking Streaming Analytics For Scale (20)

PDF
Rethinking Streaming Analytics for Scale
C4Media
 
PDF
Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson
Spark Summit
 
PPTX
Trivento summercamp masterclass 9/9/2016
Stavros Kontopoulos
 
PPTX
Functional architectural patterns
Lars Albertsson
 
PDF
Data Infrastructure for a World of Music
Lars Albertsson
 
PPT
Moving Towards a Streaming Architecture
Gabriele Modena
 
PDF
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
confluent
 
PPT
Resilience: the key requirement of a [big] [data] architecture - StampedeCon...
StampedeCon
 
PDF
Lyft data Platform - 2019 slides
Karthik Murugesan
 
PDF
The Lyft data platform: Now and in the future
markgrover
 
PDF
Streaming architecture patterns
hadooparchbook
 
PPTX
Ingestion and Dimensions Compute and Enrich using Apache Apex
Apache Apex
 
PPTX
Hadoop Summit SJ 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
PPTX
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
PPTX
Apache Apex: Stream Processing Architecture and Applications
Thomas Weise
 
PPTX
Big Data Ecosystem at LinkedIn. Keynote talk at Big Data Innovators Gathering...
Mitul Tiwari
 
PDF
Data pipelines from zero to solid
Lars Albertsson
 
PPTX
Next Gen Big Data Analytics with Apache Apex
DataWorks Summit/Hadoop Summit
 
PPTX
WebAction In-Memory Computing Summit 2015
WebAction
 
PPTX
Data Architectures for Robust Decision Making
Gwen (Chen) Shapira
 
Rethinking Streaming Analytics for Scale
C4Media
 
Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson
Spark Summit
 
Trivento summercamp masterclass 9/9/2016
Stavros Kontopoulos
 
Functional architectural patterns
Lars Albertsson
 
Data Infrastructure for a World of Music
Lars Albertsson
 
Moving Towards a Streaming Architecture
Gabriele Modena
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
confluent
 
Resilience: the key requirement of a [big] [data] architecture - StampedeCon...
StampedeCon
 
Lyft data Platform - 2019 slides
Karthik Murugesan
 
The Lyft data platform: Now and in the future
markgrover
 
Streaming architecture patterns
hadooparchbook
 
Ingestion and Dimensions Compute and Enrich using Apache Apex
Apache Apex
 
Hadoop Summit SJ 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
Apache Apex: Stream Processing Architecture and Applications
Thomas Weise
 
Big Data Ecosystem at LinkedIn. Keynote talk at Big Data Innovators Gathering...
Mitul Tiwari
 
Data pipelines from zero to solid
Lars Albertsson
 
Next Gen Big Data Analytics with Apache Apex
DataWorks Summit/Hadoop Summit
 
WebAction In-Memory Computing Summit 2015
WebAction
 
Data Architectures for Robust Decision Making
Gwen (Chen) Shapira
 

Recently uploaded (20)

PDF
Credit Without Borders: AI and Financial Inclusion in Bangladesh
Ehsan Tanim
 
PDF
Consumable AI The What, Why & How for Small Teams.pdf
Vivek Sai
 
DOCX
Basics of Cloud Computing - Cloud Ecosystem
suthak11
 
PDF
UiPath Agentic Automation session 1: RPA to Agents
suhanisingh58689
 
PDF
Taming the Chaos: How to Turn Unstructured Data into Decisions
Safe Software
 
PPTX
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
PPT
What is a Computer? Input Devices /output devices
GulJan8
 
PDF
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
PDF
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
PPTX
The various Industrial Revolutions .pptx
bandileshozi3
 
PPTX
Training Program for knowledge in solar cell and solar industry
RabindranathKaibarty
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PPTX
Modernising the Digital Integration Hub
Daniel Toomey
 
PDF
A contest of sentiment analysis: k-nearest neighbor versus neural network
IAESIJAI
 
PPTX
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
PDF
CloudStack 4.21: First Look Webinar slides
ShapeBlue
 
PDF
Produktkatalog für HOBO Datenlogger, Wetterstationen, Sensoren, Software und ...
Metrics GmbH
 
PDF
Comparative analysis of machine learning models for fake news detection in so...
IAESIJAI
 
PDF
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
PDF
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
Credit Without Borders: AI and Financial Inclusion in Bangladesh
Ehsan Tanim
 
Consumable AI The What, Why & How for Small Teams.pdf
Vivek Sai
 
Basics of Cloud Computing - Cloud Ecosystem
suthak11
 
UiPath Agentic Automation session 1: RPA to Agents
suhanisingh58689
 
Taming the Chaos: How to Turn Unstructured Data into Decisions
Safe Software
 
Build Your First AI Agent with UiPath.pptx
suhanisingh58689
 
What is a Computer? Input Devices /output devices
GulJan8
 
Getting started with AI Agents and Multi-Agent Systems
Maxim Salnikov
 
Zenith AI: Advanced Artificial Intelligence
Biz Preneurs
 
The various Industrial Revolutions .pptx
bandileshozi3
 
Training Program for knowledge in solar cell and solar industry
RabindranathKaibarty
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
Modernising the Digital Integration Hub
Daniel Toomey
 
A contest of sentiment analysis: k-nearest neighbor versus neural network
IAESIJAI
 
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
CloudStack 4.21: First Look Webinar slides
ShapeBlue
 
Produktkatalog für HOBO Datenlogger, Wetterstationen, Sensoren, Software und ...
Metrics GmbH
 
Comparative analysis of machine learning models for fake news detection in so...
IAESIJAI
 
A review of recent deep learning applications in wood surface defect identifi...
IAESIJAI
 
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 

Rethinking Streaming Analytics For Scale

  • 1. RETHINKING STREAMING ANALYTICS FOR SCALE Helena Edelson 1 @helenaedelson
  • 2. @helenaedelson Who Is This Person? • VP of Product Engineering @Tuplejump • Big Data, Analytics, Cloud Engineering, Cyber Security • Committer / Contributor to FiloDB, Spark Cassandra Connector, Akka, Spring Integration 2 • @helenaedelson • github.com/helena • linkedin.com/in/helenaedelson • slideshare.net/helenaedelson
  • 4. @helenaedelson Tuplejump - Open Source github.com/tuplejump • FiloDB - part of this talk • Calliope - the first Spark-Cassandra integration • Stargate - an open source Lucene indexer for Cassandra • SnackFS - open source HDFS for Cassandra 4
  • 5. @helenaedelson What Will We Talk About • The Problem Domain • Example Context • Rethinking Architecture • We don't have to look far to look back • Streaming & Data Science • Challenging Assumptions • Revisiting the goal and the stack • Integration • Simplification 5
  • 6. @helenaedelson THE PROBLEM DOMAIN Delivering Meaning From A Flood Of Data 6 @helenaedelson
  • 7. @helenaedelson The Problem Domain Need to build scalable, fault tolerant, distributed data processing systems that can handle massive amounts of data from disparate sources, with different data structures. 7
  • 8. @helenaedelson Translation How to build adaptable, elegant systems for complex analytics and learning tasks to run as large-scale clustered dataflows 8
  • 9. @helenaedelson How Much Data Yottabyte = quadrillion gigabytes or septillion bytes We all have a lot of data • Terabytes • Petabytes... http://en.wikipedia.org/wiki/Yottabyte 9 100 trillion $ in DC fees
  • 10. @helenaedelson Delivering Meaning • Deliver meaning in sec/sub-sec latency • Disparate data sources & schemas • Billions of events per second • High-latency batch processing • Low-latency stream processing • Aggregation of historical from the stream 10
  • 11. @helenaedelson While We Monitor, Predict & Proactively Handle • Massive event spikes & bursty traffic • Fast producers / slow consumers • Network partitioning & out of sync systems • DC down • Wait, we've DDOS'd ourselves from fast streams? • Autoscale issues – When we scale down VMs how do we not lose data? 11
  • 12. @helenaedelson And stay within our AWS / Rackspace budget 12
  • 14. @helenaedelson • Track activities of international threat actor groups, nation-state, criminal or hactivist • Intrusion attempts • Actual breaches • Profile adversary activity • Analysis to understand their motives, anticipate actions and prevent damage Adversary Profiling & Hunting: Online & Offline 14
  • 15. @helenaedelson • Machine events • Endpoint intrusion detection • Anomalies/indicators of attack or compromise • Machine learning • Training models based on patterns from historical data • Predict potential threats • profiling for adversary Identification Stream Processing 15
  • 16. @helenaedelson Data Requirements & Description • Streaming event data • Log messages • User activity records • System ops & metrics data • Disparate data sources • Wildly differing data structures 16
  • 17. @helenaedelson Massive Amounts Of Data • One machine can generate 2+ TB per day • Tracking millions of devices • 1 million writes per second - bursty • High % writes, lower % reads 17
  • 19. @helenaedelson 19 few years in Silicon Valley Cloud Engineering team @helenaedelson
  • 20. @helenaedelson 20 Batch analytics data flow from several years ago looked like...
  • 21. @helenaedelson 21 Batch analytics data flow from several years ago looked like...
  • 23. @helenaedelson 23 Sweet, let's triple the code we have to update and regression test every time our analytics logic changes
  • 25. @helenaedelson Streaming: Big Data, Fast Data, Fast Timeseries Data • Reactive processing of data as it comes in to derive instant insights • Is this enough? • Need to combine with existing big data, historical processing, ad hoc queries 25
  • 26. @helenaedelson New Requirements, Common Use Case I need fast access to historical data on the fly for predictive modeling with real time data from the stream 26
  • 27. @helenaedelson It's Not A Stream It's A Flood • Netflix • 50 - 100 billion events per day • 1 - 2 million events per second at peak • LinkedIn • 500 billion write events per day • 2.5 trillion read events per day • 4.5 million events per second at peak with Kafka • 1 PB of stream data 27
  • 28. @helenaedelson Which Translates To • Do it fast • Do it cheap • Do it at scale 28
  • 30. @helenaedelson AND THEN WE GREEKED OUT 30 Lambda
  • 31. @helenaedelson Lambda Architecture A data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream processing methods. 31 • Or, "How to beat the CAP theorum" • An approach coined by Nathan Mars • This was a huge stride forward
  • 32. @helenaedelson • Doing complex asynchronous transformations • That need to run with low latency (say, a few seconds to a few hours) • Examples • Weather analytics and prediction system • News recommendation system 32 Applications Using Lambda Architecture
  • 34. @helenaedelson Implementing Is Hard • Real-time pipeline backed by KV store for updates • Many moving parts - KV store, real time, batch • Running similar code in two places • Still ingesting data to Parquet/HDFS • Reconcile queries against two different places 34
  • 35. @helenaedelson Performance Tuning & Monitoring on so many disparate systems 35 Also Hard
  • 36. @helenaedelson 36 λ: Streaming & Batch Flows Evolution Or Just Addition? Or Just Technical Debt?
  • 37. @helenaedelson Lambda Architecture Ingest an immutable sequence of records is captured and fed into • a batch system • and a stream processing system in parallel 37
  • 39. @helenaedelson Which Translates To • Performing analytical computations & queries in dual systems • Duplicate Code • Untyped Code - Strings • Spaghetti Architecture for Data Flows • One Busy Network 39
  • 40. @helenaedelson 40 Why? • Why support code, machines and running services of two analytics systems? • Is a separate batch system needed? • Can we do everything in a streaming system?
  • 41. @helenaedelson YES 41 • A unified system for streaming and batch • Real-time processing and reprocessing • Code changes • Fault tolerance http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html - Jay Kreps
  • 43. @helenaedelson Extract, Transform, Load (ETL) • Extraction of data from one system into another • Transforming it • Loading it into another system 43
  • 44. @helenaedelson ETL • Each step can introduce errors and risk • Writing intermediary files • Parsing and re-parsing plain text • Tools can cost millions of dollars • Decreases throughput • Increased complexity • Can duplicate data after failover 44
  • 45. @helenaedelson Extract, Transform, Load (ETL) "Designing and maintaining the ETL process is often considered one of the most difficult and resource- intensive portions of a data warehouse project." http://docs.oracle.com/cd/B19306_01/server.102/b14223/ettover.htm 45 Also unnecessarily redundant and often typeless
  • 46. @helenaedelson And let's duplicate the pattern over all our DataCenters 46
  • 47. @helenaedelson 47 These are not the solutions you're looking for
  • 49. @helenaedelson Removing The 'E' in ETL Thanks to technologies like Avro and Protobuf we don’t need the “E” in ETL. Instead of text dumps that you need to parse over multiple systems: E.g Scala and Avro • A return to strong typing in the big data ecosystem • Can work with binary data that remains strongly typed 49
  • 50. @helenaedelson Removing The 'L' in ETL If data collection is backed by a distributed messaging system (e.g. Kafka) you can do real-time fanout of the ingested data to all consumers. No need to batch "load". • From there each consumer can do their own transformations 50
  • 53. @helenaedelson Pick Technologies Wisely Based on your requirements • Latency • Real time / Sub-Second: < 100ms • Near real time (low): > 100 ms or a few seconds - a few hours • Consistency • Highly Scalable • Topology-Aware & Multi-Datacenter support • Partitioning Collaboration - do they play together well 53
  • 54. @helenaedelson And Remember • Flows erode • Entropy happens • "Everything fails, all the time" - Kyle Kingsbury 54
  • 56. @helenaedelson Stream Processing & Frameworks 56 + GearPump
  • 57. @helenaedelson Strategies 57 • Partition For Scale & Data Locality • Replicate For Resiliency • Share Nothing • Fault Tolerance • Asynchrony • Async Message Passing • Memory Management • Data lineage and reprocessing in runtime • Parallelism • Elastically Scale • Isolation • Location Transparency
  • 58. @helenaedelson Fault Tolerance • Graceful service degradation • Data integrity / accuracy under failure • Resiliency during traffic spikes • Pipeline congestion / bottlenecks • Easy to debug and find failure source • Easy to deploy 58
  • 59. @helenaedelson Strategy Technologies Scalable Infrastructure / Elastic Spark, Cassandra, Kafka Partition For Scale, Network Topology Aware Cassandra, Spark, Kafka, Akka Cluster Replicate For Resiliency Spark,Cassandra, Akka Cluster all hash the node ring Share Nothing, Masterless Cassandra, Akka Cluster both Dynamo style Fault Tolerance / No Single Point of Failure Spark, Cassandra, Kafka Replay From Any Point Of Failure Spark, Cassandra, Kafka, Akka + Akka Persistence Failure Detection Cassandra, Spark, Akka, Kafka Consensus & Gossip Cassandra & Akka Cluster Parallelism Spark, Cassandra, Kafka, Akka Asynchronous Data Passing Kafka, Akka, Spark Fast, Low Latency, Data Locality Cassandra, Spark, Kafka Location Transparency Akka, Spark, Cassandra, Kafka My Nerdy Chart 59
  • 60. @helenaedelson SMACK • Scala & Spark Streaming • Mesos • Akka • Cassandra • Kafka 60
  • 61. @helenaedelson Spark Streaming • One runtime for streaming and batch processing • Join streaming and static data sets • No code duplication • Easy, flexible data ingestion from disparate sources to disparate sinks • Easy to reconcile queries against multiple sources • Easy integration of KV durable storage 61
  • 62. @helenaedelson Training Data Feature Extraction Model Training Model Testing Test Data Your Data Extract Data To Analyze Train your model to predict 62 val context = new StreamingContext(conf, Milliseconds(500)) val model = KMeans.train(dataset, ...) // learn offline val stream = KafkaUtils .createStream(ssc, zkQuorum, group,..) .map(event => model.predict(event.feature))
  • 63. @helenaedelson 63 High performance concurrency framework for Scala and Java • Fault Tolerance • Asynchronous messaging and data processing • Parallelization • Location Transparency • Local / Remote Routing • Akka: Cluster / Persistence / Streams
  • 64. @helenaedelson Akka Actors 64 A distribution and concurrency abstraction • Compute Isolation • Behavioral Context Switching • No Exposed Internal State • Event-based messaging • Easy parallelism • Configurable fault tolerance
  • 65. @helenaedelson High Performance Streaming Built On Akka • Apache Flink - uses Akka for • Actor model and hierarchy, Deathwatch and distributed communication between job and task managers • GearPump - models the entire streaming system with an actor hierarchy • Supervision, Isolation, Concurrency 65
  • 66. @helenaedelson Apache Cassandra • Extremely Fast • Extremely Scalable • Multi-Region / Multi-Datacenter • Always On • No single point of failure • Survive regional outages • Easy to operate • Automatic & configurable replication 66
  • 67. @helenaedelson 90% of streaming data at Netflix is stored in Cassandra 67
  • 69. @helenaedelson 69 KillrWeather http://github.com/killrweather/killrweather A reference application showing how to easily integrate streaming and batch data processing with Apache Spark Streaming, Apache Cassandra, Apache Kafka and Akka for fast, streaming computations on time series data in asynchronous event-driven environments. http://github.com/databricks/reference-apps/tree/master/timeseries/scala/timeseries-weather/src/main/scala/com/ databricks/apps/weather
  • 70. @helenaedelson val context = new StreamingContext(conf, Seconds(1)) val stream = KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder]( context, kafkaParams, kafkaTopics) stream.flatMap(func1).saveToCassandra(ks1,table1) stream.map(func2).saveToCassandra(ks1,table1) context.start() 70 Kafka, Spark Streaming and Cassandra
  • 71. @helenaedelson class KafkaProducerActor[K, V](config: ProducerConfig) extends Actor {
 
 override val supervisorStrategy =
 OneForOneStrategy(maxNrOfRetries = 10, withinTimeRange = 1.minute) {
 case _: ActorInitializationException => Stop
 case _: FailedToSendMessageException => Restart case _: ProducerClosedException => Restart case _: NoBrokersForPartitionException => Escalate case _: KafkaException => Escalate
 case _: Exception => Escalate
 } 
 private val producer = new KafkaProducer[K, V](producerConfig)
 
 override def postStop(): Unit = producer.close() 
 def receive = {
 case e: KafkaMessageEnvelope[K,V] => producer.send(e)
 }
 } 71 Kafka, Spark Streaming, Cassandra & Akka
  • 72. @helenaedelson Spark Streaming, ML, Kafka & C* val ssc = new StreamingContext(new SparkConf()…, Seconds(5)
 val testData = ssc.cassandraTable[String](keyspace,table).map(LabeledPoint.parse)
 
 val trainingStream = KafkaUtils.createStream[K, V, KDecoder, VDecoder](
 ssc, kafkaParams, topicMap, StorageLevel.MEMORY_ONLY) .map(_._2).map(LabeledPoint.parse) trainingStream.saveToCassandra("ml_keyspace", "raw_training_data")
 
 val model = new StreamingLinearRegressionWithSGD()
 .setInitialWeights(Vectors.dense(weights))
 .trainOn(trainingStream) //Making predictions on testData model .predictOnValues(testData.map(lp => (lp.label, lp.features))) .saveToCassandra("ml_keyspace", "predictions") 72
  • 75. @helenaedelson 75 class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 } Kafka, Spark Streaming, Cassandra & Akka
  • 76. @helenaedelson class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 } 76 Now we can replay • On failure • Reprocessing on code changes • Future computation...
  • 77. @helenaedelson 77 Here we are pre-aggregating to a table for fast querying later - in other secondary stream aggregation computations and scheduled computing class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 }
  • 78. @helenaedelson CREATE TABLE weather.raw_data (
 wsid text, year int, month int, day int, hour int, 
 temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, one_hour_precip PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); CREATE TABLE daily_aggregate_precip (
 wsid text,
 year int,
 month int,
 day int,
 precipitation counter,
 PRIMARY KEY ((wsid), year, month, day)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC); Data Model (simplified) 78
  • 79. @helenaedelson 79 class KafkaStreamingActor(params: Map[String, String], ssc: StreamingContext) extends AggregationActor(settings: Settings) {
 import settings._ 
 val stream = KafkaUtils.createStream(
 ssc, params, Map(KafkaTopicRaw -> 1), StorageLevel.DISK_ONLY_2)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw)
 
 stream .map(hour => (hour.wsid, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 } Gets the partition key: Data Locality Spark C* Connector feeds this to Spark Cassandra Counter column in our schema, no expensive `reduceByKey` needed. Simply let C* do it: not expensive and fast.
  • 80. @helenaedelson The Thing About S3 "Amazon S3 is a simple key-value store" - docs.aws.amazon.com/AmazonS3/latest/dev/UsingObjects.html • Keys 2015/05/01 and 2015/05/02 do not live in the “same place” • You can roll your own with AmazonS3Client and do the heavy lifting yourself and throw that data into Spark 80
  • 81. @helenaedelson CREATE TABLE weather.raw_data (
 wsid text, year int, month int, day int, hour int, 
 temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, one_hour_precip PRIMARY KEY ((wsid), year, month, day, hour)
 ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); C* Clustering Columns Writes by most recent Reads return most recent first Timeseries Data 81 Cassandra will automatically sort by most recent for both write and read
  • 82. @helenaedelson 82 val multipleStreams = for (i <- numstreams) { streamingContext.receiverStream[HttpRequest](new HttpReceiver(port)) } streamingContext.union(multipleStreams) .map { httpRequest => TimelineRequestEvent(httpRequest)} .saveToCassandra("requests_ks", "timeline") CREATE TABLE IF NOT EXISTS requests_ks.timeline ( timesegment bigint, url text, t_uuid timeuuid, method text, headers map <text, text>, body text, PRIMARY KEY ((url, timesegment) , t_uuid) ); Record Every Event In The Order In Which It Happened, Per URL timesegment protects from writing unbounded partitions. timeuuid protects from simultaneous events over-writing one another.
  • 83. @helenaedelson val stream = KafkaUtils.createDirectStream(...)
 .map(_._2.split(","))
 .map(RawWeatherData(_)) 
 stream.saveToCassandra(CassandraKeyspace, CassandraTableRaw) stream .map(hour => (hour.id, hour.year, hour.month, hour.day, hour.oneHourPrecip))
 .saveToCassandra(CassandraKeyspace, CassandraTableDailyPrecip)
 83 Replay and Reprocess - Any Time Data is on the nodes doing the querying - Spark C* Connector - Partitions • Timeseries data with Data Locality • Co-located Spark + Cassandra nodes • S3 does not give you Cassandra & Spark Streaming: Data Locality For Free®
  • 84. @helenaedelson class PrecipitationActor(ssc: StreamingContext, settings: Settings) extends AggregationActor {
 import akka.pattern.pipe 
 def receive : Actor.Receive = { 
 case GetTopKPrecipitation(wsid, year, k) => topK(wsid, year, k, sender)
 }
 
 /** Returns the 10 highest temps for any station in the `year`. */
 def topK(wsid: String, year: Int, k: Int, requester: ActorRef): Unit = {
 val toTopK = (aggregate: Seq[Double]) => TopKPrecipitation(wsid, year,
 ssc.sparkContext.parallelize(aggregate).top(k).toSeq)
 
 ssc.cassandraTable[Double](keyspace, dailytable)
 .select("precipitation")
 .where("wsid = ? AND year = ?", wsid, year)
 .collectAsync().map(toTopK) pipeTo requester
 }
 } 84 Queries pre-aggregated tables from the stream Compute Isolation: Actor
  • 85. @helenaedelson 85 class TemperatureActor(sc: SparkContext, settings: Settings) extends AggregationActor {
 import akka.pattern.pipe 
 def receive: Actor.Receive = {
 case e: GetMonthlyHiLowTemperature => highLow(e, sender)
 }
 
 def highLow(e: GetMonthlyHiLowTemperature, requester: ActorRef): Unit =
 sc.cassandraTable[DailyTemperature](keyspace, daily_temperature_aggr)
 .where("wsid = ? AND year = ? AND month = ?", e.wsid, e.year, e.month)
 .collectAsync()
 .map(MonthlyTemperature(_, e.wsid, e.year, e.month)) pipeTo requester } C* data is automatically sorted by most recent - due to our data model. Additional Spark or collection sort not needed. Efficient Batch Analysis
  • 87. @helenaedelson Everything On The Streaming Platform 87
  • 88. @helenaedelson Reprocessing • Start a new stream job to re-process from the beginning • Save re-processed data as a version table • Application should then read from new version table • Stop old version of the job, and delete the old table 88
  • 89. @helenaedelson One Pipeline For Fast & Big Data 89 • How do I make the SMACK stack work for ML, Ad-Hoc + Fast Data? • How do I combine Spark Streaming + Ad Hoc and have good performance?
  • 90. @helenaedelson FiloDB Designed to ingest streaming data, including machine, event, and time-series data, and run very fast analytical queries over them. 90 • Distributed, versioned, columnar analytics database • Built for fast streaming analytics & OLAP • Currently based on Apache Cassandra & Spark • github.com/tuplejump/FiloDB
  • 91. @helenaedelson Breakthrough Performance For Analytical Queries • Queries run in parallel in Spark for scale-out ad-hoc analysis • Fast for interactive data science and ad hoc queries • Up to 200x Faster Queries for Spark on Cassandra 2.x • Parquet Performance with Cassandra Flexibility • Increased performance ceiling coming 91
  • 92. @helenaedelson Versioning & Why It Matters • Immutability • Databases: let's mutate one giant piece of state in place • Basically hasn't changed since 1970's! • With Big Data and streaming, incremental processing is increasingly important 92
  • 93. @helenaedelson FiloDB Versioning FiloDB is built on functional principles and lets you version and layer changes • Incrementally add a column or a few rows as a new version • Add changes as new versions, don't mutate! • Writes are idempotent - exactly once ingestion • Easily control what versions to query • Roll back changes inexpensively • Stream out new versions as continuous queries :) 93
  • 94. @helenaedelson No Cassandra? Keep All In Memory 94 • Unlike RDDs and DataFrames, FiloDB can ingest new data, and still be fast • Unlike RDDs, FiloDB can filter in multiple ways, no need for entire table scan
  • 95. @helenaedelson Spark Streaming to FiloDB val ratingsStream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder(
 ssc, kafkaParams, topics)
 
 ratingsStream.foreachRDD { (message: RDD[(String, String)], batchTime: Time) =>
 val df = message
 .map(_._2.split(","))
 .map(rating => Rating(trim(rating))
 .toDF("fromuserid", "touserid", "rating")
 
 // add the batch time to the DataFrame
 val dfWithBatchTime = df.withColumn(
 "batch_time", org.apache.spark.sql.functions.lit(batchTime.milliseconds))
 
 // save the DataFrame to FiloDB
 dfWithBatchTime.write.format("filodb.spark")
 .option("dataset", "ratings")
 .save()
 } 95 .dfWithBatchTime.write.format("org.apache.spark.sql.cassandra")
  • 96. @helenaedelson Architectyr? "This is a giant mess" - Going Real-time - Data Collection and Stream Processing with Apache Kafka, Jay Kreps 96