Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark

Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark

• 1.
  Arbitrary Stateful Aggregations usingStructured Streaming in Apache Spark™ Burak Yavuz 5/16/2017
• 2.
  2 Outline • Structured StreamingConcepts • Stateful Processing in Structured Streaming • Use Cases • Demos
• 3.
  3 The simplest wayto perform streaming analytics is not having to reason about streaming at all
• 4.
  4
• 5.
  5 Input: data fromsource as an append-only table Trigger: how frequently to check input for new data Query: operations on input usual map/filter/reduce new window, session ops Trigger: every 1 sec 1 2 3 Time data up to 1 Input data up to 2 data up to 3 Query New Model
• 6.
  6 Trigger: every 1sec 1 2 3 result for data up to 1 Result Query Time data up to 1 Input data up to 2 result for data up to 2 data up to 3 result for data up to 3 Output [complete mode] output all the rows in the result table New Model Result: final operated table updated every trigger interval Output: what part of result to write to data sink after every trigger Complete output: Write full result table every time
• 7.
  7 Trigger: every 1sec 1 2 3 result for data up to 1 Result Query Time data up to 1 Input data up to 2 result for data up to 2 data up to 3 result for data up to 3 Output [append mode] output only new rows since last trigger Result: final operated table updated every trigger interval Output: what part of result to write to data sink after every trigger Complete output: Write full result table every time Append output: Write only new rows that got added to result table since previous batch *Not all output modes are feasible with all queries New Model
• 8.
  8
• 9.
  9 Output Modes • Appendmode (default) - New rows added to the Result Table since the last trigger will be outputted to the sink. Rows will be output only once, and cannot be rescinded. Example use cases: ETL
• 10.
  10 Output Modes • Completemode - The whole Result Table will be outputted to the sink after every trigger. This is supported for aggregation queries. Example use cases: Monitoring
• 11.
  11 Output Modes • Updatemode - (Available since Spark 2.1.1) Only the rows in the Result Table that were updated since the last trigger will be outputted to the sink. Example use cases: Alerting, Sessionization
• 12.
  12 Outline • Structured StreamingConcepts • Stateful Processing in Structured Streaming • Use Cases • Demos
• 13.
  13 Event time Aggregations Manyuse cases require aggregate statistics by event time E.g. what's the #errors in each system in 1 hour windows? Many challenges Extracting event time from data, handling late, out-of-order data DStream APIs were insufficient for event time operations
• 14.
  14 Event time Aggregations Windowingis just another type of grouping in Struct. Streaming number of records every hour parsedData .groupBy(window("timestamp","1  hour")) .count() parsedData .groupBy( "device",   window("timestamp","10  mins")) .avg("signal") avg signal strength of each device every 10 mins Use built-in functions to extract event-time No need for separate extractors
• 15.
  15 Advanced Aggregations Powerful built-in aggregations Multiplesimultaneous aggregations Custom aggs using reduceGroups, UDAFs parsedData .groupBy(window("timestamp","1  hour")) .agg(avg("signal"),  stddev("signal"),  max("signal")) variance,  stddev,  kurtosis,  stddev_samp,  collect_list,   collect_set,  corr,  approx_count_distinct,  ...   //  Compute  histogram  of  age  by  name. val hist =  ds.groupBy(_.type).mapGroups { case (type,  data:  Iter[DeviceData])  => val buckets =  new Array[Int](10)             data.map(_.signal).foreach {  a  => buckets(a/10)+=1 }         (type,  buckets) }
• 16.
  16 Stateful Processing forAggregations In-memory, streaming state maintained for aggregations 12:00 - 13:00 1 12:00 - 13:00 3 13:00 - 14:00 1 12:00 - 13:00 3 13:00 - 14:00 2 14:00 - 15:00 5 12:00 - 13:00 5 13:00 - 14:00 2 14:00 - 15:00 5 15:00 - 16:00 4 12:00 - 13:00 3 13:00 - 14:00 2 14:00 - 15:00 6 15:00 - 16:00 4 16:00 - 17:00 3 13:00 14:00 15:00 16:00 17:00 Keeping state allows late data to update counts of old windows But size of the state increases indefinitely if old windows not dropped red = state updated with late data
• 17.
  17
• 18.
  18 Watermarking and LateData Watermark [Spark 2.1] - a moving threshold that trails behind the max seen event time Trailing gap defines how late data is expected to be event time max event time watermark data older than watermark not expected 12:30 PM 12:20 PM trailing gap of 10 mins
• 19.
  19 Watermarking and LateData Data newer than watermark may be late, but allowed to aggregate Data older than watermark is "too late" and dropped State older than watermark automatically deleted to limit the amount of intermediate state max event time event time watermark late data allowed to aggregate data too late, dropped
• 20.
  20 Watermarking and LateData max event time event time watermark allowed lateness of 10 mins parsedData .withWatermark("timestamp",  "10  minutes") .groupBy(window("timestamp","5  minutes")) .count() late data allowed to aggregate data too late, dropped Control the tradeoff between state size and lateness requirements Handle more late à keep more state Reduce state à handle less lateness
• 21.
  21 Watermarking to LimitState [Spark 2.1] data too late, ignored in counts, state dropped Processing Time12:00 12:05 12:10 12:15 12:10 12:15 12:20 12:07 12:13 12:08 EventTime 12:15 12:18 12:04 watermark updated to 12:14 - 10m = 12:04 for next trigger, state < 12:04 deleted data is late, but considered in counts parsedData .withWatermark("timestamp",  "10  minutes") .groupBy(window("timestamp","5  minutes")) .count() system tracks max observed event time 12:08 wm = 12:04 10min 12:14 More details in blog post!
• 22.
  22
• 23.
  23 Working With Time df.withWatermark("timestampColumn", "5  hours") .groupBy(window("timestampColumn",  "1  minute")) .count() .writeStream .trigger("10  seconds") Separate processing details (output rate, late data tolerance) from query semantics.
• 24.
  24 Working With Time df.withWatermark("timestampColumn", "5  hours") .groupBy(window("timestampColumn",  "1  minute")) .count() .writeStream .trigger("10  seconds") How to group data by time Same in streaming & batch
• 25.
  25 Working With Time df.withWatermark("timestampColumn", "5  hours") .groupBy(window("timestampColumn",  "1  minute")) .count() .writeStream .trigger("10  seconds") How late data can be
• 26.
  26 Working With Time df.withWatermark("timestampColumn", "5  hours") .groupBy(window("timestampColumn",  "1  minute")) .count() .writeStream .trigger("10  seconds") How often to emit updates
• 27.
  27 Arbitrary Stateful Operations[Spark 2.2] mapGroupsWithState allows any user-defined stateful ops to a user-defined state Direct support for per-key timeouts in event-time or processing-time supports Scala and Java ds.groupByKey(groupingFunc) .mapGroupsWithState (timeoutConf) (mappingWithStateFunc) def mappingWithStateFunc( key: K,   values: Iterator[V],   state: GroupState[S]): U =  {   //  update  or  remove  state //  set  timeouts //  return  mapped  value }
• 28.
  28 flatMapGroupsWithState • Applies thegiven function to each group of data, while maintaining a user-defined per-group state • Invoked once per group in batch • Invoked each trigger (with the existence of data) per group in streaming • Requires user to provide an output mode for the function
• 29.
  29 flatMapGroupsWithState • mapGroupsWithState isa special case with • Output mode: Update • Output size: 1 row per group • Supports both Processing Time and Event Time timeouts
• 30.
  30 Outline • Structured StreamingConcepts • Stateful Processing in Structured Streaming • Use Cases • Demos
• 31.
  31 Alerting val monitoring  = stream .as[Event] .groupBy(_.id) .flatMapGroupsWithState(Append,  GST.ProcessingTimeTimeout)  { (id:  Int,  events:  Iterator[Event],  state:  GroupState[…])  => ... } .writeStream .queryName("alerts") .foreach(new  PagerdutySink(credentials)) Monitor a stream using custom stateful logic with timeouts.
• 32.
  32 Sessionization val monitoring  = stream .as[Event] .groupBy(_.session_id) .mapGroupsWithState(GST.EventTimeTimeout)  { (id:  Int,  events:  Iterator[Event],  state:  GroupState[…])  => ... } .writeStream .parquet("/user/sessions") Analyze sessions of user/system behavior
• 33.
  33 Demo
• 34.
  34 SPARK SUMMIT 2017 DATASCIENCE AND ENGINEERING AT SCALE JUNE 5 – 7 | MOSCONE CENTER | SAN FRANCISCO ORGANIZED BY spark-summit.org/2017 Discount Code: Databricks
• 35.
  We are hiring! https://databricks.com/company/careers
• 36.
  Thank You “Does anyonehave any questions for my answers?” - Henry Kissinger