Managing multiple event types in a single topic with Schema Registry | Bill Bejeck, Confluent
Download as pptx, pdf0 likes3,557 views
The document discusses best practices for managing multiple event types within Apache Kafka topics, emphasizing the use of schemas for compatibility and modularity. It covers approaches to defining subjects and strategies for mapping events to topics, including the importance of maintaining order for related events. Additionally, it provides code examples for handling events using Avro and Protobuf in Kafka Streams.
Managing multiple event types in a single topic with Schema Registry | Bill Bejeck, Confluent
- 1. Multiple Events in a single topic Bill Bejeck, Integration Architect
- 2. @bbejeck Nice to meet you! • Integration Architect on DevX team • Prior to DevX ~3 years as engineer on Kafka Streams team • Apache Kafka® Committer • Author of “Kafka Streams in Action” - 2nd edition underway! 2
- 3. Event objects form a contract
- 4. @bbejeck Kafka brokers work with bytes only 4
- 5. @bbejeck Kafka brokers work with bytes only 5
- 6. @bbejeck Potentially make unexpected changes 6
- 7. @bbejeck Potentially make unexpected changes 7
- 17. @bbejeck What is a Subject ? • Defines a namespace for a schema • Compatibility checks are per subject • When a schema evolves, the subject remains, but the schema gets a new ID and version • Different approaches – subject name strategies 17
- 21. Mapping events to topics
- 22. @bbejeck Mapping events to topics • Conventional wisdom – topic per distinct event 22 Purchase Customer Interaction
- 25. @bbejeck Mapping events to topics • For related events the order is important – one topic • But we want to make sure we can limit the different record types • What are our options? 25
- 27. @bbejeck Schema references • A field in a schema has an object represented by a registered schema • Allows for schema re-use and modularity • Enhances multi-event topics by enabling TopicNameStrategy (restricts the number of types) 27
- 33. @bbejeck Schema Refs redux - TopicNameStrategy 33
- 34. How to handle ?
- 35. @bbejeck Avro (Consumer) 35 records = consumer.poll(Duration.ofSeconds(5)); records.forEach(record -> { final SpecificRecord avroRecord = record.value(); if (avroRecord instanceof PageView) { PageView pageView = (PageView) action; ..... } else if (avroRecord instanceof Purchase) { Purchase purchase = (Purchase) action; ..... } });
- 36. @bbejeck Protobuf (Consumer) 36 records = protoConsumer.poll(Duration.ofSeconds(5)); records.forEach(record -> { final CustomerEventProto.CustomerEvent customerEvent = record.value(); ActionCase actionCase = customerEvent.getActionCase(); switch (actionCase) { case PURCHASE -> ...; case PAGE_VIEW -> ...; case ACTION_NOT_SET -> ...; } });
- 37. @bbejeck Protobuf (Consumer) 37 records = protoConsumer.poll(Duration.ofSeconds(5)); records.forEach(record -> { final CustomerEventProto.CustomerEvent customerEvent = record.value(); customerEvent = record.value(); if (customerEvent.hasPageView()) { ... } else if (customerEvent.hasPurchase()) { ... } });
- 38. @bbejeck Kafka Streams 38 public CustomerInfo transform(String readOnlyKey, SpecificRecord value){ CustomerInfo customerInfo = store.get(readOnlyKey); if (customerInfo == null) { customerInfo = CustomerInfo.newBuilder() .setCustomerId(readOnlyKey) .build(); }
- 39. @bbejeck Kafka Streams 39 if(value instanceof PageView) { PageView pageView = (PageView) value; customerInfo.getPageViews().add(pageView.getUrl()); } else if (value instanceof Purchase) { Purchase purchase = (Purchase) value; customerInfo.getItems().add(purchase.getItem()); } //Details left out
- 40. @bbejeck Resources • Martin Kleppmann - https://www.confluent.io/blog/put-several-event-types-kafka-topic/ • Robert Yokota - https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka- topic/ • Documentation - https://docs.confluent.io/platform/current/schema- registry/index.html#sr-overview • Examples - https://github.com/bbejeck/multiple-events-kafka-summit-europe-2021 40
Editor's Notes
- #5: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #6: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #7: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #8: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #9: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #10: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #11: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #12: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #13: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #14: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #15: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #16: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #19: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #20: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #21: SubjectNameStrategy, it's basically a function of the form (topic-name, record-schema) -> subject-name. TopicNameStrategy ignores the record-schema, RecordNameStrategy ignores the topic-name, and TopicRecordNameStrategy uses both. By using TopicNameStrategy (ignoring the record-schema), it allows all records in the topic to be constrained by a single subject (as opposed to an indeterminate number of subjects with the other strategies)
- #24: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #25: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #29: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #30: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #31: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #32: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #33: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #34: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #36: Kafka Streams has 2 APIs DSL and a Processor API This is the same streams application you saw earlier so we won’t go into too much detail about this program
- #37: Kafka Streams has 2 APIs DSL and a Processor API This is the same streams application you saw earlier so we won’t go into too much detail about this program
- #38: Kafka Streams has 2 APIs DSL and a Processor API This is the same streams application you saw earlier so we won’t go into too much detail about this program
- #39: Kafka Streams has 2 APIs DSL and a Processor API This is the same streams application you saw earlier so we won’t go into too much detail about this program
- #40: Kafka Streams has 2 APIs DSL and a Processor API This is the same streams application you saw earlier so we won’t go into too much detail about this program
- #43: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.
- #44: Kafka Streams has 2 APIs DSL and a Processor API This is the same streams application you saw earlier so we won’t go into too much detail about this program
- #45: Discuss what a log is – a file where records are appended to the end of the file What does a distributed log mean What is a topic – a topic is a log with the topic name is the directory name and the file is the log with messages Discuss the offsets picture – Kafka assigns each incoming record with a number, offset, which is its ordinal position in the file.