[DataSciCon] Divide, distribute and conquer stream v. batch
1 like356 views
The document discusses the differences between batch and stream processing, highlighting the importance of time models and event-based semantics in data processing. It emphasizes the challenges of handling out-of-order and late data, as well as providing real-time computation results through windowing techniques. Additionally, it touches on the Lambda architecture and offers examples such as predicting flight arrivals using stream processing.
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://image.slidesharecdn.com/dividedistributeandconquer-streamv-171201225205/85/DataSciCon-Divide-distribute-and-conquer-stream-v-batch-21-320.jpg)
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://image.slidesharecdn.com/dividedistributeandconquer-streamv-171201225205/85/DataSciCon-Divide-distribute-and-conquer-stream-v-batch-22-320.jpg)
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://image.slidesharecdn.com/dividedistributeandconquer-streamv-171201225205/85/DataSciCon-Divide-distribute-and-conquer-stream-v-batch-34-320.jpg)
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://image.slidesharecdn.com/dividedistributeandconquer-streamv-171201225205/85/DataSciCon-Divide-distribute-and-conquer-stream-v-batch-35-320.jpg)
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://image.slidesharecdn.com/dividedistributeandconquer-streamv-171201225205/85/DataSciCon-Divide-distribute-and-conquer-stream-v-batch-36-320.jpg)
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://image.slidesharecdn.com/dividedistributeandconquer-streamv-171201225205/85/DataSciCon-Divide-distribute-and-conquer-stream-v-batch-37-320.jpg)
More Related Content
![[Philly JUG] Divide, Distribute and Conquer: Stream v. Batch](https://cdn.slidesharecdn.com/ss_thumbnails/dividedistributeandconquer-streamv-170920171540-thumbnail.jpg?width=560&fit=bounds)
Similar to [DataSciCon] Divide, distribute and conquer stream v. batch (20)
![[Philly ETE] Java Puzzlers NG](https://cdn.slidesharecdn.com/ss_thumbnails/javapuzzlerss02-phillyete-170418150118-thumbnail.jpg?width=560&fit=bounds)
![[JBreak] Блеск И Нищета Распределенных Стримов - 04-04-2017](https://cdn.slidesharecdn.com/ss_thumbnails/gamov-jbreak-04-04-2017-170404164301-thumbnail.jpg?width=560&fit=bounds)
![[OracleCode - SF] Distributed caching for your next node.js project](https://cdn.slidesharecdn.com/ss_thumbnails/distributedcachingforyournextnode-170302181006-thumbnail.jpg?width=560&fit=bounds)
![[OracleCode SF] In memory analytics with apache spark and hazelcast](https://cdn.slidesharecdn.com/ss_thumbnails/in-memoryanalyticswithapachesparkandhazelcast-oraclecode-03-01-2017-170302180618-thumbnail.jpg?width=560&fit=bounds)
![[Jfokus] Riding the Jet Streams](https://cdn.slidesharecdn.com/ss_thumbnails/ridingthejetstreams-jfokus-02-08-2017-170208120631-thumbnail.jpg?width=560&fit=bounds)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://cdn.slidesharecdn.com/ss_thumbnails/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616-thumbnail.jpg?width=560&fit=bounds)
![[Codemash] Caching Made "Bootiful"!](https://cdn.slidesharecdn.com/ss_thumbnails/cachingmadebootifulcodemash11217-170112212321-thumbnail.jpg?width=560&fit=bounds)
![[JokerConf] Верхом на реактивных стримах, 10/13/2016](https://cdn.slidesharecdn.com/ss_thumbnails/ridingthejetstreams-jokerconf-10-13-2016-161014221528-thumbnail.jpg?width=560&fit=bounds)
[DataSciCon] Divide, distribute and conquer stream v. batch
- 1. DIVIDE, DISTRIBUTE AND CONQUER: STREAM V. BATCH
- 3. Who am I?
- 6. Solutions Architect Developer Advocate @gamussa in internetz Who am I?
- 7. Solutions Architect Developer Advocate @gamussa in internetz Hey you, yes, you, go follow me in twitter © Who am I?
- 8. @gamussa @confluentinc @DataSciCon BATCH PROCESSING Data at rest
- 9. @gamussa @confluentinc @DataSciCon Data and Queries Origin and processing
- 13. @gamussa @confluentinc @DataSciCon ✓ … inherently immutable Data… ✓ … time-based
- 14. @gamussa @confluentinc @DataSciCon CRUD -> CR
- 15. @gamussa @confluentinc @DataSciCon Processing is a query
- 16. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set
- 17. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set Projection
- 18. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set Projection Aggregations
- 19. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set Projection Aggregations Joins
- 20. @gamussa @confluentinc @DataSciCon Lambda architecture origins http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
- 23. @gamussa @confluentinc @DataSciCon https://mapr.com/developercentral/lambda-architecture/ Lambda Architecture
- 25. @gamussa @confluentinc @DataSciCon TFW Trying to explain modern big data landscape
- 27. @gamussa @confluentinc @DataSciCon STREAM PROCESSING Data is motion
- 28. @gamussa @confluentinc @DataSciCon Streaming Platform
- 29. @gamussa @confluentinc @DataSciCon Streaming Platform
- 31. @gamussa @confluentinc @DataSciCon Interesting cases Before You Go
- 32. I FOUND YOUR LACK OF FAULT TOLERANCE DISTURBING
- 33. Data is too important to store it in one computer
- 38. @gamussa @confluentinc @DataSciCon How to process «infinite» data?
- 39. @gamussa @confluentinc @DataSciCon Time model
- 40. @gamussa @confluentinc @DataSciCon Time model Different use cases time semantics
- 41. @gamussa @confluentinc @DataSciCon Time model Different use cases time semantics Majority of use cases require event- time semantics
- 42. @gamussa @confluentinc @DataSciCon Time model Different use cases time semantics Majority of use cases require event- time semantics Other use cases may require processing-time or special variants like ingestion-time
- 43. @gamussa @confluentinc @DataSciCon Time Model
- 44. @gamussa @confluentinc @DataSciCon Time Model
- 45. @gamussa @confluentinc @DataSciCon Time Model
- 46. @gamussa @confluentinc @DataSciCon Windowing Input data, where colors represent different users events Rectangles denote different event-time windows processing-time event-time windowing alice bob dave
- 48. @gamussa @confluentinc @DataSciCon Windowing Windowing is an operation that groups events Most commonly needed: time windows, session windows Examples: ✗Real-time monitoring: 5-minute averages ✗Reader behavior on a website: user browsing sessions
- 49. @gamussa @confluentinc @DataSciCon Out-of-order and late data Is very common in practice, not a rare corner case ✗Related to time model discussion
- 50. @gamussa @confluentinc @DataSciCon Out-of-order and late data
- 51. @gamussa @confluentinc @DataSciCon Out-of-order and late data Users with mobile phones enter airplane, lose Internet connectivity
- 52. @gamussa @confluentinc @DataSciCon Out-of-order and late data Users with mobile phones enter airplane, lose Internet connectivity Emails are being written during the 10h flight
- 53. @gamussa @confluentinc @DataSciCon Out-of-order and late data Users with mobile phones enter airplane, lose Internet connectivity Emails are being written during the 10h flight Internet connectivity is restored, phones will send queued emails now
- 54. @gamussa @confluentinc @DataSciCon Stream Processing: results
- 55. @gamussa @confluentinc @DataSciCon Stream Processing: results • Yes, it’s possible to get computation results in real time
- 56. @gamussa @confluentinc @DataSciCon Stream Processing: results • Yes, it’s possible to get computation results in real time • Windows – finite view of infinite data • Based on temporal characteristics of the evet
- 57. @gamussa @confluentinc @DataSciCon Stream Processing: results • Yes, it’s possible to get computation results in real time • Windows – finite view of infinite data • Based on temporal characteristics of the evet • Late event processing • You choose how long to wait
- 58. @gamussa @confluentinc @DataSciCon DEMO Let’s analyze flights
- 60. @gamussa @confluentinc @DataSciCon Example: Training Flight Prediction Model