Delivering: from Kafka to WebSockets | Adam Warski, SoftwareMill

Delivering: from Kafka to WebSockets | Adam Warski, SoftwareMill

• 1.
  @adamwarski, Kafka SummitEurope, May 2021 Delivering: from Kafka to WebSockets
• 2.
  Problem statement Send messagesfrom a Kafka topic to WebSockets service service service Kafka WebSocket node Clustered, multi-node 1 node? Multiple nodes?
• 3.
  Adding more nodes service service service Kafka WebSocket node Howto connect? WebSocket node
• 4.
  A Kafka topichas many partitions service service service WebSocket node How to connect? WebSocket node Partition 1 Partition 3 Partition 4 Partition 2
• 5.
  About • 14 yearsof backend engineering • mainly using Scala, but quite a lot of Kafka as well • coder & CTO @ SoftwareMill • custom software: distributed systems, Big&Fast Data, ML/AI, Blockchain, Messaging, … • Scala/Kafka/Cassandra consulting • Blog: http://www.softwaremill.com/blog • Twitter: https://twitter.com/adamwarski
• 6.
  #1: everybody readseverything WebSocket node WebSocket node Partition 1 Partition 3 Partition 4 Partition 2
• 7.
  #2: intelligent router service service service 1 9 2 4 3 8 6 WebSocket node Partition basedon client id WebSocket node Partition 1 Partition 3 Partition 4 Partition 2 Router
• 8.
  #2: intelligent router service service service 1 9 2 4 3 8 6 WebSocket node Break existing connections WebSocket node Partition1 Partition 3 Partition 4 Partition 2 WebSocket node Router
• 9.
  #3: redistributors WebSocket node Backchannel - client assignment WebSocket node Partition1 Partition 3 Partition 4 Partition 2 redistributor redistributor
• 10.
  #3.1: redistributors backchannelw/ Kafka WebSocket node report client state WebSocket node redistributor redistributor Kafka build an in- memory view
• 11.
  #3.2: redistributors backchannelw/ cache WebSocket node report client state WebSocket node redistributor redistributor memcached / Redis / … query where client is located
• 12.
  #3: redistributors WebSocket node WebSocket node Partition 1 Partition3 Partition 4 Partition 2 redistributor redistributor partition reassignment
• 13.
  #3.A: forwarding usingKafka WebSocket node WebSocket node Partition 1 Partition 3 Partition 4 Partition 2 redistributor redistributor WS partition 1 WS partition 2 custom partition assignment
• 14.
  #3.B: forwarding usingHTTP WebSocket node WebSocket node Partition 1 Partition 3 Partition 4 Partition 2 redistributor redistributor
• 15.
  How is Kafkainfluencing our design? • Static & limited partitions • no partitions per client • no partition per WS node (or hard to implement) • for good reasons!
• 16.
  Summing up • Threepossible solutions: • everybody reads everything • intelligent router • redistributor service service service WebSocket node How to connect? WebSocket node
• 17.
  Summing up: redistributor •Backchannel • using Kafka • using an in-memory cache • Forwarding • using Kafka • using HTTP service service service WebSocket node How to connect? WebSocket node
• 18.
  Links • https://kafka.apache.org • Real-timedata pipelines and streaming ebook:
 https://kafka.softwaremill.com • Evaluating persistent, replicated message queues: 
 https://softwaremill.com/mqperf/ • Twitter: https://twitter.com/adamwarski
• 19.
  Thank you! @adamwarski