Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
4 likes4,983 views
This document provides an overview of spark-timeseries, an open source time series library for Apache Spark. It discusses the library's design choices around representing multivariate time series data, partitioning time series data for distributed processing, and handling operations like lagging and differencing on irregular time series data. It also presents examples of using the library to test for stationarity, generate lagged features, and perform Holt-Winters forecasting on seasonal passenger data.
![Vectors Date/
Time
Series 1 Series 2
Vector 1 2:30:01 4.56 78.93
Vector 2 2:30:02 4.57 79.92
Vector 3 2:30:03 4.87 79.91
Vector 4 2:30:04 4.48 78.99
RDD[(ZonedDateTime, Vector)]
DateTime
Index
Vector for
Series 1
Vector for
Series 2
2:30:01 4.56 78.93
2:30:02 4.57 79.92
2:30:03 4.87 79.91
2:30:04 4.48 78.99
TimeSeriesRDD(
DateTimeIndex,
RDD[Vector]
)
Columnar representation Row-based representation](https://image.slidesharecdn.com/2asimonouellette-170214185944/85/Time-Series-Analytics-with-Spark-Spark-Summit-East-talk-by-Simon-Ouellette-6-320.jpg)
![Current design
Assumption: a single time series must
fit inside executory memory!
TimeSeriesRDD (
DatetimeIndex,
RDD[(K, Vector)]
)
IrregularDatetimeIndex (
Array[Long], // Other limitation: Scala arrays = 232 elements
java.time.ZoneId
)](https://image.slidesharecdn.com/2asimonouellette-170214185944/85/Time-Series-Analytics-with-Spark-Spark-Summit-East-talk-by-Simon-Ouellette-15-320.jpg)
Time Series Analytics with Spark: Spark Summit East talk by Simon Ouellette
- 1. Time Series Analytics with Spark Simon Ouellette Faimdata
- 2. What is spark-timeseries? • Open source time series library for Apache Spark 2.0 • Sandy Ryza – Advanced Analytics with Spark: Patterns for Learning from Data at Scale – Senior Data Scientist at Clover Health • Started in February 2015 https://github.com/sryza/spark-timeseries
- 3. Who am I? • Chief Data Science Officer at Faimdata • Contributor to spark-timeseries since September 2015 • Participated in early design discussions (March 2015) • Been an active user for ~2 years http://faimdata.com
- 4. Survey: Who uses time series?
- 5. Design Question #1: How do we structure multivariate time series? Columnar or Row-based?
- 6. Vectors Date/ Time Series 1 Series 2 Vector 1 2:30:01 4.56 78.93 Vector 2 2:30:02 4.57 79.92 Vector 3 2:30:03 4.87 79.91 Vector 4 2:30:04 4.48 78.99 RDD[(ZonedDateTime, Vector)] DateTime Index Vector for Series 1 Vector for Series 2 2:30:01 4.56 78.93 2:30:02 4.57 79.92 2:30:03 4.87 79.91 2:30:04 4.48 78.99 TimeSeriesRDD( DateTimeIndex, RDD[Vector] ) Columnar representation Row-based representation
- 7. Columnar vs Row-based • Lagging • Differencing • Rolling operations • Feature generation • Feature selection • Feature transformation • Regression • Clustering • Classification • Etc. More efficient in columnar representation: More efficient in row-based representation:
- 8. Example: lagging operation • Time complexity: O(N) (assumes pre-sorted RDD) • For each row, we need to get values from previous k rows • Time complexity: O(K) • For each column to lag, we truncate most recent k values, and truncate the DateTimeIndex’s oldest k values. Columnar representationRow-based representation
- 9. Example: regression • We’re estimating: • The lagged values are typically part of each row, because they are pre-generated as new features. • Stochastic Gradient Descent: we iterate on examples and estimate error gradient to adjust weights, which means that we care about rows, not columns. • To avoid shuffling, the partitioning must be done such that all elements of a row are together in the same partition (so the gradient can be computed locally).
- 10. Current solution • Core representation is columnar. • Utility functions to go to/from row-based. • Reasoning: spark-timeseries operations are mostly time-related, i.e. columnar. Row-based operations are about relationships between the variables (ML/statistical), thus external to spark- timeseries.
- 11. Typical time series analytics workflow:
- 12. Survey: Who uses univariate time series that don’t fit inside a single executor’s memory? (or multivariate of which a single variable’s time series doesn’t fit)
- 13. Design Question #2: How do we partition the multi-variate time series for distributed processing? Across features, or across time?
- 14. Current design Assumption: a single time series must fit inside executory memory!
- 15. Current design Assumption: a single time series must fit inside executory memory! TimeSeriesRDD ( DatetimeIndex, RDD[(K, Vector)] ) IrregularDatetimeIndex ( Array[Long], // Other limitation: Scala arrays = 232 elements java.time.ZoneId )
- 16. Future improvements • Creation of a new TimeSeriesRDD-like class that will be longitudinally (i.e. across time) partitioned rather than horizontally (i.e. across features). • Keep both types of partitioning, on a case-by- case basis.
- 17. Design Question #3: How do we lag, difference, etc.? Re-sampling, or index-preserving?
- 18. Option #1: re-sampling Irregular Time y value at t x value at t 1:30:05 51.42 4.87 1:30:07.86 52.37 4.99 1:30:07.98 53.22 4.95 1:30:08.04 55.87 4.97 1:30:12 54.84 5.12 1:30:14 49.88 5.10 Uniform Time y value at t x value at (t – 1) 1:30:06 51.42 4.87 1:30:07 51.42 4.87 1:30:08 53.22 4.87 1:30:09 55.87 4.95 1:30:10 55.87 4.97 1:30:11 55.87 4.97 1:30:12 54.84 4.97 1:30:13 54.84 5.12 1:30:14 49.88 5.12 Before After (1 second lag)
- 19. Option #2: index preserving Irregular Time y value at t x value at t 1:30:05 51.42 4.87 1:30:07.86 52.37 4.99 1:30:07.98 53.22 4.95 1:30:08.04 55.87 4.97 1:30:12 54.84 5.12 1:30:14 49.88 5.10 Irregular Time y value at t x value at (t – 1) 1:30:05 51.42 N/A 1:30:07.86 52.37 4.87 1:30:07.98 53.22 4.87 1:30:08.04 55.87 4.87 1:30:12 54.84 4.97 1:30:14 49.88 5.12 Before After (1 second lag)
- 20. Current functionality • Option #1: resample() function for lagging/differencing by upsampling/downsampling. – Custom interpolation function (used when downsampling) • Conceptual problems: – Information loss and duplication (downsampling) – Bloating (upsampling)
- 21. Current functionality • Option #2: functions to lag/difference irregular time series based on arbitrary time intervals. (preserves index) • Same thing: custom interpolationfunction can be passed for when downsampling occurs.
- 22. Overview of current API
- 23. High-level objects • TimeSeriesRDD • TimeSeries • TimeSeriesStatisticalTests • TimeSeriesModel • DatetimeIndex • UnivariateTimeSeries
- 24. TimeSeriesRDD • collectAsTimeSeries • filterStartingBefore, filterStartingAfter, slice • filterByInstant • quotients, differences, lags • fill: fills NaNs by specified interpolation method (linear, nearest, next, previous, spline, zero) • mapSeries • seriesStats: min, max, average, std. deviation • toInstants, toInstantsDataFrame • resample • rollSum, rollMean • saveAsCsv, saveAsParquetDataFrame
- 25. TimeSeriesStatisticalTests • Stationarity tests: – Augmented Dickey-Fuller (adftest) – KPSS (kpsstest) • Serial auto-correlation tests: – Durbin-Watson (dwtest) – Breusch-Godfrey (bgtest) – Ljung-Box (lbtest) • Breusch-Pagan heteroskedasticity test (bptest) • Newey-West variance estimator (neweyWestVarianceEstimator)
- 26. TimeSeriesModel • AR, ARIMA • ARX, ARIMAX (i.e. with exogenous variables) • Exponentially weighted moving average • Holt-winters method (triple exp. smoothing) • GARCH(1,1),ARGARCH(1,1,1)
- 27. Others • Java bindings • Python bindings • YAHOO financial data parser
- 28. Code example #1 Time Y X 12:45:01 3.45 25.0 12:46:02 4.45 30.0 12:46:58 3.45 40.0 12:47:45 3.00 35.0 12:48:05 4.00 45.0 Y is stationary X is integrated of order 1
- 29. Code example #1
- 30. Code example #1 Time y d(x) Lag1(y) Lag2(y) Lag1(d(x)) Lag2(d(x)) 12:45:01 3.45 12:46:02 4.45 5.0 3.45 12:46:58 3.45 10.0 4.45 3.45 5.0 12:47:45 3.00 -5.0 3.45 4.45 10.0 5.0 12:48:05 4.00 10.0 3.00 3.45 -5.0 10.0
- 31. Code example #2 • We will use Holt-Winters to forecast some seasonal data. • Holt-winters: exponential moving average applied to level, trend and seasonal component of the time series, then combined into global forecast.
- 33. Code example #2
- 34. Code example #2 350 400 450 500 550 600 650 1 2 3 4 5 6 7 8 9 10 11 12 13 Holt-Winters forecast validation (additive & multiplicative) Actual Additive forecast Multiplicativeforecast