Redis Streams for Event-Driven Microservices
Download as PPTX, PDF1 like1,944 views
- Bobby Calderwood is a software engineer and founder who will discuss using event sourcing with Redis and Clojure(Script) to build distributed systems. - He worked with a major automotive manufacturer on a project using this architecture including event sourcing on Redis streams, implemented in Clojure(Script), to address their needs for asynchronous processing, data replay/redo, and sharing data between teams. - The presentation will include a demo of building a sample coffee shop ordering system using this architecture.
More Related Content
Similar to Redis Streams for Event-Driven Microservices (20)
More from Redis Labs (20)
Redis Streams for Event-Driven Microservices
- 1. PRESENTED BY
- 2. PRESENTED BY Hello, I’m Bobby Calderwood! Software Engineer @Cognitect for 4+ years, Consulting and Datomic Distinguished Engineer @CapitalOne for 3+ years, helping redesign and move mission-critical banking systems to the cloud Founder @Evident_Systems, helping smaller banks redesign and move mission- critical banking systems to the cloud Meet with me! http://meet.evident.systems/founder
- 3. PRESENTED BY Agenda: 1 Story of a real customer system Event-Sourcing + Redis + Clojure(Script) = Joy 2 But why that architecture? A quick primer on CQRS + Event Sourcing using Redis Streams and Clojure(Script) 3 Demo system Let’s pretend we have to build a system, and then modify it with an asynchronous integration
- 4. PRESENTED BY Major automobile manufacturer Greenfield system for new capability, political proving ground for stack and process Highly asynchronous image processing and significant UI display/manipulation Possible need for data re-processing/re-play, undo/redo, audit, and sharing with other teams We decided to architect it using Event Sourcing on Streaming Data using Redis, and implemented in Clojure(Script) Solving a customer problem
- 5. PRESENTED BY But why that architecture?!?
- 6. PRESENTED BY At very least, we knew we had to process images asynchronously We suspected that we’d need to keep this pipeline flexible, to try different processing steps and techniques We knew we’d need to convey the results of this pipeline back to the UI (the oft-forgotten participant in the distributed system) To facilitate our asynchronous processing pipeline
- 7. PRESENTED BY Integration among “object/entity-oriented” microservices becomes complex and burdensome • Balkanization of data stores • Data “owner” determines data access pattern • Non-determinism == impossibility of reasoning about system: – state – causality/time – modes of failure Because HTTP APIs aren’t enough
- 9. PRESENTED BY Request/response protocols face a fundamental epistemological problem • What is happening, anything new? • When should I check for new data? • Who’s responsible to tell me? Because HTTP APIs aren’t enough
- 10. PRESENTED BY Event Sourcing provides “as-of” consistency across distributed participants (Online Event Processing) Streaming Data protocols convey new data/events to interested participants (subscribers) Any participant can simulate synchrony and consistency by blocking and waiting for a specific event Event Sourcing + Streaming Data Facilitate Distributed Consistency Image by Alan Light CC BY-SA 3.0
- 11. PRESENTED BY Cross-cutting concerns need access to your application’s data • Analytics • Audit/Compliance/Governance • Systems Integration And they don’t want to ETL out of your database… • Lossy, due to your app’s worldview and assumptions • Latent • Requires coordination with you (“we’ll add it to the backlog”) • Complex Event Sourcing + Streaming Data Enable Cooperation without Coordination
- 12. PRESENTED BY Only one way to cause change to a bounded context • Send a Command to its API Two ways to read from a bounded context: • Listen to its Event stream – create your own Read Models from one or more Event Streams – or maybe incorporate subset of Events into your own Event stream • Ask Queries of its Read Models CQRS Separates Action from Perception
- 15. PRESENTED BY Why Redis? (for streaming data)
- 16. PRESENTED BY Redis is ubiquitous: you (or your customer) already has it in the stack Redis is battle-tested, and Ops is comfortable with it Redis is (often) easier to operate than Kafka (with some tradeoffs) Redis is Already There, and Just Works
- 17. PRESENTED BY After Kafka popularized the stream data structure, Salvatore noticed gap in Redis, as he described: http://antirez.com/news/114 Existing data structures got almost there • Sorted Sets: memory hungry, clients can’t block awaiting new records, order not stable • Lists: linear time to scan, can’t seek to specific position, no fan-out to multiple consumers • Pub/Sub: doesn’t maintain history, can’t do windowed aggregation, etc. Streams maintain queryable history, and efficiently convey to subscribers Redis Provides Streams
- 18. PRESENTED BY XADD — Produce to a Stream XRANGE, XREVRANGE — Query stream per range of existing offsets, non- blocking, batch-ish XREAD — Optionally-blocking read (conveys novelty), realtime-ish XREADGROUP — Optionally-blocking read by a group of consumers, ack-ing messages to advance consumer offset (but watch message ordering!) But How to Redis Stream? https://redis.io/topics/streams-intro
- 20. PRESENTED BY Clojure is a fantastic all-purpose programming language whose functional programming philosophy aligns well with event sourcing, streaming data We needed to build all components quickly, with a small team of Clojurists: • UI in ClojureScript – React + Reagent + re-frame – Access to JavaScript libraries with Clojure language semantics • HTTP API in Clojure + Pedestal (+ GraphQL via Lacinia) • Workers in Clojure Clojure’s core.async library provide in-process semantics aligned with streaming data Clojure(Script) is powerful and full-stack
- 21. PRESENTED BY Very solid, simple Redis client Generated from Redis artifacts Stays close to Redis semantics https://github.com/ptaoussanis/carmine/ Clojure has Carmine
- 22. PRESENTED BY Let’s Build a Demo System!
- 23. PRESENTED BY Store Front Web/Mobile App How Customers order online, pick up in store Easy enough, but… Baristas have to make the drinks How to coordinate their work? Should we notify the customers when their order is ready? Later we discover that managers must monitor inventory levels! Curiouser and Curiouser! A Coffee Shop
- 25. PRESENTED BY Demo System In Action
- 26. PRESENTED BY Build APIs from an immutable log of business events Using CQRS + Event Sourcing via Redis Streams https://redis.io/topics/streams-intro Online Event Processing, Martin Kleppmann et. al. https://queue.acm.org/detail.cfm?id=3321612 Try Clojure and ClojureScript https://clojure.org/ Check out the demo app! https://github.com/bobby/redisconf19-demo Conclusion
Editor's Notes
- #12: "We'll add it to the backlog" is just the polite way of saying no. —Mark Dalgleish https://twitter.com/markdalgleish/status/1143411813465223168