Apache Kafka Fundamentals for Architects, Admins and Developers

Apache Kafka Fundamentals for Architects, Admins and Developers

• 1.
  1 Mid-Ohio Data UsersGroup Meetup Organizer: Derek Kane Host: Navigator Management Partners
• 2.
  2 Apache Kafka® Fundamentals for Architects,Admins and Developers
• 3.
  3 About Me ● PatrickDruley ● Senior Systems Engineer @ Confluent (1yr) ● Formerly at Oracle and Teradata ● Ohioan, currently in Medina Ohio ● Github: https://github.com/pdruley ● Twitter: @PatrickLovesAK
• 4.
  4
• 5.
  5 A Modern, DistributedPlatform for Data Streams
• 6.
  6 Apache Kafka® ismade up of distributed, immutable, append-only commit logs
• 7.
  7 Intelligent Clients andNaive Pipes Writers Kafka cluster Readers
• 8.
  88 Scalability of afilesystem • hundreds of MB/s throughput • many TB per server • commodity hardware
• 9.
  99 Guarantees of aDatabase • Strict ordering • Persistence
• 10.
  1010 Rewind & Replay Resetto any point in the shared narrative
• 11.
  1111 Distributed by design •Replication • Fault Tolerance • Partitioning • Elastic Scaling
• 12.
  1212 Kafka Topics my-topic my-topic-partition-0 my-topic-partition-1 my-topic-partition-2 broker-1 broker-2 broker-3
• 13.
  1313 Partition Leadership andReplication Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
• 14.
  1414 Partition Leadership andReplication - node failure Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
• 15.
  1515 Demo Time! https://github.com/pdruley/public_notebooks/blob/master/Glorious_Kafka_Command_Line_Utilites.ipynb
• 16.
  1616 Resources Free E-Books fromConfluent! I Heart Logs: https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/ Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/ Confluent Blog: https://www.confluent.io/blog Confluent Podcast: https://podcasts.apple.com/us/podcast/streaming-audio-a-confluent-podcast-about-apache-kafka/id1 401509765?mt=2 Thank You! patrick@confluent.io @PatrickLovesAK
• 17.
  1717 Thank You!