A visual introduction to Apache Kafka

Editor's Notes

• #3: What is Kafka? Kafka is a distributed streams processing system, it allows distributed producers to send messages to distributed consumers via a Kafka cluster.
• #4: Kafka benefits Fast – high throughput and low latency Scalable – horizontally scalable, just add nodes and partitions Reliable – distributed and fault tolerant Zero data loss – messages are persisted to disk with immutable log Open Source – An Apache project Available as a Managed Service - on multiple cloud platforms
• #5: Kafka benefits Fast – high throughput and low latency Scalable – horizontally scalable, just add nodes and partitions Reliable – distributed and fault tolerant Zero data loss – messages are persisted to disk with immutable log Open Source – An Apache project Available as a Managed Service - on multiple cloud platforms
• #6: Back to the intro Kafka overview diagram, it’s a bit monochrome and boring. This talk will be more colourful and it’s going to be an extended story…
• #7: Back to the intro Kafka overview diagram, it’s a bit monochrome and boring. This talk will be more colourful and it’s going to be an extended story…
• #8: Let’s build a modern day fully electronic postal service – the Kafka Postal Service
• #9: What does a postal service do? It sends messages from A to B (animated with sounds, click!)
• #10: A is a producer, sends a message…
• #11: To B, the consumer.
• #12: Actually, no. Due to the decline in “snail mail” volumes, direct deliveries have been cancelled.
• #13: Actually, no. Due to the decline in “snail mail” volumes, direct deliveries have been cancelled.
• #14: Instead we have “Poste Restante” - Not a post office in a restaurant It’s called general delivery (in the US). The mail is delivered to a post office, and they hold it for you until you call for it.
• #15: Consumers poll for messages by visiting the counter at the post office.
• #16: Consumers poll for messages by visiting the counter at the post office.
• #17: Kafka topics act like a Post Office. What are the benefits? Disconnected delivery – consumer doesn’t need to be available to receive messages Less effort for the messaging service – only has to delivery to a few locations not larger number of consumer addresses Can scale better and handle more complex delivery semantics!
• #18: Kafka topics act like a Post Office. What are the benefits? Disconnected delivery – consumer doesn’t need to be available to receive messages Less effort for the messaging service – only has to delivery to a few locations not larger number of consumer addresses Can scale better and handle more complex delivery semantics!
• #19: First lets see how it scales. What if there are many consumers for a topic?
• #20: A single counter introduces delays and limits concurrency
• #21: More counters increases concurrency and can reduce delays
• #22: Kafka Topics have 1 or more Partitions, partitions function like multiple counters and enable high concurrency.
• #23: Before looking at delivery semantics what does a message actually look like? In Kafka a message is called a Record and is like a letter.
• #24: The topic is the destination.
• #25: The “Postmark” includes a timestamp, offset in the topic, and the partition it was sent to. Time semantics are flexible, either the time of event creation, ingestion, or processing.
• #26: The “Postmark” includes a timestamp, offset in the topic, and the partition it was sent to. Time semantics are flexible, either the time of event creation, ingestion, or processing.
• #27: There’s also a thing called a Key, which is is optional. It refines the destination so it’s a bit like the rest of the address. We want this letter sent to Santa not just any Elf.
• #28: There’s also a thing called a Key, which is is optional. It refines the destination so it’s a bit like the rest of the address. We want this letter sent to Santa not just any Elf.
• #29: The value is the contents (just a byte array). Kafka Producers and consumers need to have a shared serializer and de-serializer for both the key and value.
• #30: The value is the contents (just a byte array). Kafka Producers and consumers need to have a shared serializer and de-serializer for both the key and value.
• #31: Kafka doesn’t look inside the value, but the Producer and Consumer can, and the Consumer can try and make sense of the message… I wonder what Santa will be delivering?
• #32: Next lets look at delivery semantics. For example, do we care if the message actually arrives or not?
• #33: Yes we do! Guaranteed message delivery is desirable. Homing pigeons got lost or eaten, so need to send the message with multiple pigeons
• #34: Yes we do! Guaranteed message delivery is desirable. Homing pigeons got lost or eaten, so need to send the message with multiple pigeons
• #35: Yes we do! Guaranteed message delivery is desirable. Homing pigeons got lost or eaten, so need to send the message with multiple pigeons
• #36: Yes we do! Guaranteed message delivery is desirable. Homing pigeons got lost or eaten, so need to send the message with multiple pigeons
• #37: How does Kafka guarantee delivery? A Message (M1) is written to a broker (2)
• #38: How does Kafka guarantee delivery? A Message (M1) is written to a broker (2)
• #39: and the message is always persisted to disk.
• #40: This makes it resilient to loss of power.
• #41: The message is also replicated on multiple “brokers” , 3 is typical
• #42: And makes it resilient to loss of most servers
• #43: Finally the producer gets acknowledgement once the message is persisted and replicated (configurable for number, and sync or async) The message is now available from more than one broker in case some fail. This also increases the read concurrency as partitions are spread over multiple brokers.
• #44: Finally the producer gets acknowledgement once the message is persisted and replicated (configurable for number, and sync or async) The message is now available from more than one broker in case some fail. This also increases the read concurrency as partitions are spread over multiple brokers.
• #45: Now let’s look at the 2nd aspect of delivery semantics. Who gets the messages and how many times are messages delivered? Kafka is “pub-sub”, It’s loosely coupled, producers and consumers don’t know about each other
• #46: Now let’s look at the 2nd aspect of delivery semantics. Who gets the messages and how many times are messages delivered? Kafka is “pub-sub”, It’s loosely coupled, producers and consumers don’t know about each other
• #47: Filtering, or which consumers get which messages, is topic based Publishers send messages to topics Consumers subscribe to topics of interest, e.g. parties. When they poll they only receive messages sent to those topics.
• #48: Filtering, or which consumers get which messages, is topic based Publishers send messages to topics Consumers subscribe to topics of interest, e.g. parties. When they poll they only receive messages sent to those topics.
• #49: Filtering, or which consumers get which messages, is topic based Publishers send messages to topics Consumers subscribe to topics of interest, e.g. parties. When they poll they only receive messages sent to those topics.
• #50: Filtering, or which consumers get which messages, is topic based Publishers send messages to topics Consumers subscribe to topics of interest, e.g. parties. When they poll they only receive messages sent to those topics.
• #51: Just a few more details and we can see how this works. Partitions and consumer groups enable sharing of work across multiple consumers, the more partitions a topic has the more consumers it supports
• #52: Kafka supports delivery of the same message to multiple consumers. Kafka doesn’t throw messages away immediately they are delivered, so the same message can easily be delivered to multiple consumer groups.
• #53: Consumers subscribed to ”parties” topic are allocated partitions. When they poll they will only get messages from their allocated partitions.
• #54: This enables consumers in the same group to share the work around. Each consumer gets only a subset of the available messages.
• #55: Multiple groups enable message broadcasting. Messages are duplicated across groups, as each consumer group receives a copy of each message.
• #56: Which messages are delivered to which consumers? The final aspect of delivery semantics is to do with message keys. If a message has a key, then Kafka uses Partition based delivery. Messages with the same key are always sent to the same partition and therefore the same consumer. And the order is guaranteed.
• #57: If the key is null, then Kafka uses round robin delivery. Each message is delivered to the next partition.
• #58: Let’s look at an example with 2 consumer groups. Nerds, which has multiple consumers, and Hairy which has a single consumer.
• #59: Let’s look at an example with 2 consumer groups. Nerds, which has multiple consumers, and Hairy which has a single consumer.
• #60: Let’s look at an example with 2 consumer groups. Nerds, which has multiple consumers, and Hairy which has a single consumer.
• #61: Looking at the case where there’s No Key 1st, each message (1, 2, etc) is sent to the next partition, and all consumers allocated to that partition will receive the message when they poll next.
• #62: Here’s what actually happens. We’re not showing the producer or topic for simplicity. You’ll have to imagine them.
• #63: Both Groups subscribe to Topic “parties” (11 partitions, so 1 consumer per partition).
• #64: 2 Producer sends record with the value “Cool pool party - Invitation” to “parties” topic (there’s no key)
• #65: 3 Bill and Chewbacca receive a copy of the invitation and plan to attend
• #66: 4. Producer sends another record with the value “Cool pool party – Cancelled” to “parties” topic
• #67: In the Nerds group, Paul gets the message this time as it’s round robin, and Chewy gets it as he’s the only consumer in his group. Paul ignores it as he didn’t get the original invite Bill wastes his time trying to go The rest of the gang aren’t surprised at not receiving any invites and stay home to do some hacking Chewy plans something else fun instead...
• #68: A visit to the hairdressers!
• #69: How does it work if there is a Key? The key is hashed to a partition, and the Message is sent to that partition. Assume there are 3 messages, and messages 1 and 2 are hashed to same partition.
• #70: Here’s what happens with a key, assuming that the key is the “title” of the message (“Cool pool party”) As before Both Groups subscribe to Topic “parties” (11 partitions). Producer sends record <key=“Cool pool party”, value=“Invitation”> to “parties” topic
• #71: Here’s what happens with a key, assuming that the key is the “title” of the message (“Cool pool party”) As before Both Groups subscribe to Topic “parties” (11 partitions). Producer sends record <key=“Cool pool party”, value=“Invitation”> to “parties” topic
• #72: As before, Bill and Chewbacca receive a copy of the invitation and plan to attend
• #73: 4. Producer sends another record with the same key but a cancellation value to “parties” topic
• #74: This time, Bill and Chewbacca receive the cancellation (the same consumers as the key is identical)
• #75: Producer sends out another invitation to a Halloween party. The key is different this time.
• #76: Paul receives the Halloween invitation as the key is different and the record is sent to the partition that Paul is allocated Chewy is the only consumer in his group so he gets every record no matter what partition it’s sent to
• #77: This time Chewy gets dressed up and goes to the party.
• #78: Who did you imagine was producing the invitations? Maybe this fellow.
• #80: Here’s a UML-like diagram with the main Kafka components. We’ve introduced Producers, Topics, Partitions, Consumer Groups and Consumes today. There’s a lot more to explore, including how Kafka provides replication, and the Connect and Streaming APIs.
• #81: To finish up here are three important use cases for Kafka
• #82: The Real-time Data pipeline features Ingestion of multiple heterogeneous sources Sending data to multiple heterogeneous sinks Acts as a buffer to smooth out load spikes Enables use cases which reprocess data (e.g. disaster recovery)
• #83: Real-time Event processing features: Simple event driven applications (If X then Y…) May write and read from other data sources (e.g. Cassandra) New Events sent back to Kafka or to other systems E.g. Anomaly Detection, check out my current blog series if you are interested in this example.
• #84: Streams processing features: Complex streams processing (multiple events and streams) Time, windows, and transformations Streams API, includes state store This a visualization of the streams topology from the streams processing pipeline from my previous blog series, Kongo, a Kafka IoT logistics application. It continuously computes the loads for trucks and checks if they are overloaded.
• #85: Here’s a real example from Linkedin, who developed Kafka. Before Kafka they had spaghetti integration of monolithic applications. To accommodate growing membership and increasing site complexity, they migrated from a monolithic application infrastructure to one based on microservices, which made the integration even more complex.
• #86: After Kafka Rather than maintaining and scaling each pipeline individually, they invested in the development of a single, distributed pub-sub platform. Thus, Kafka was born. This main benefit was better Service decoupling and independent scaling.