More Related Content
Similar to Apache Kafka - Patterns anti-patterns (20)
Apache Kafka - Patterns anti-patterns
- 1. #DevoxxFR Apache Kafka Patterns / AntiPatterns Florent Ramière @framiere Jean-Louis Boudart @jlboudart 1
- 2. 2 Problem ?
- 3. 3 Silos explained by Data Gravity concept As data accumulates (builds mass) there is a greater likelihood that additional services and applications will be attracted to this data. This is the same effect gravity has on objects around a planet. As the mass or density increases, so does the strength of gravitational pull.
- 4. 4 With
- 5. 5 How
- 6. 6 Store & ETL Process Publish & Subscribe In short
- 7. 7 From a simple idea
- 8. 8 From a simple idea
- 9. 9 with great properties ! • Scalability • Retention • Durability • Replication • Security • Resiliency • Throughput • Ordering • Exactly Once Semantic • Transaction • Idempotency • Immutability • …
- 11. 11 11 What could potentially go wrong ?
- 12. 12
- 13. 13 13 …which is true for any data systems
- 14. 14 14 Not thinking about Durability
- 15. 15 Data durability If you didn’t think about it… it’s not durable!
- 16. 16
- 17. 17
- 18. 18
- 19. 19 19 And you might have lost data!
- 20. 20 Data durability Kafka is not waiting for a disk flush by default. Durability is achieved through replication.
- 21. 21
- 22. 22
- 23. 23
- 24. 24
- 25. 25 Data durability It depends on your configuration...
- 26. 26
- 27. 27
- 28. 28
- 29. 29
- 30. 30 Data durability acks=1 (default value) good for latency acks=all good for durability
- 31. 31
- 32. 32 acks=all The leader will wait for the full set of in-sync replicas to acknowledge the record.
- 33. 33
- 34. 34
- 35. 35
- 36. 36 min.insync.replicas minimum number of replicas that must acknowledge. Default is 1.
- 37. 37
- 38. 38 38 Data Durability while Producing ? Tune it with the parameters acks and min.insync.replicas
- 39. 39 defaults The default values are optimized for availability & latency. If durability is more important, tune it!
- 40. 40 40 Deploying on multi datacenters ?
- 41. 41
- 42. 42 Multi-dc It’s quite complicated... It’s easy to make it wrong on many levels. It could be a 3h talk.
- 43. 43 Multi-dc Disaster recovery for multi datacenter
- 44. 44 44 What about the consumers ?
- 45. 45 consumers Consumer can read only committed data.
- 46. 46
- 47. 47 47 Think about data durability and decide of the best trade-off for you
- 48. 48 Throughput, latency, durability, availability Optimizing your Apache Kafka deployment
- 49. 49 49 Focusing only on the happy path
- 50. 50
- 51. 51
- 52. 52
- 53. 53 retries It will cause the client to resend any record whose send fails with a potentially transient error. Default value : 0
- 54. 54
- 55. 55 retries Use built in retries ! Bump it from 0 to infinity!
- 56. 56 retries But you are exposed to a different kind of issue…
- 57. 57
- 58. 58 enable.idempotence When set to 'true', the producer will ensure that exactly one copy of each message is written. Default value: false
- 59. 59
- 60. 60
- 61. 61 61 Use built in idempotency!
- 62. 62 62 But it does not save you from - Managing exception and failure - Developing Idempotent consumer
- 64. 64
- 65. 65 65 At least once (default) At most once Exactly Once
- 66. 66
- 67. 67
- 68. 68
- 69. 69
- 70. 70
- 71. 71
- 72. 72 commit Manually committing aggressively... Add a huge workload on Apache Kafka
- 73. 73
- 74. 74 commit Manually committing aggressively... Does not provide exactly once semantic
- 75. 75 75 Embrace at least once
- 76. 76 76 Rely on Kafka Streams with Exactly Once !
- 78. 78
- 79. 79 Future<RecordMetadata> send(ProducerRecord<K, V> record);
- 80. 80 Future<RecordMetadata> send(ProducerRecord<K, V> record, Callback callback);
- 81. 81 producer.send(record, (metadata, exception) -> { });
- 82. 82 error handling We don’t expect the unexpected until the unexpected is expected.
- 83. 83
- 84. 84 error handling A message can not be processed
- 85. 85 error handling A message can not be processed A message doesn’t have the expected schema
- 86. 86 86 Retry
- 87. 87
- 89. 89
- 90. 90 90 Write to a dead letter queue and continue
- 91. 91
- 93. 93
- 95. 95 95 Handle the exceptions ! https://eng.uber.com/reliable-reprocessing/
- 97. 97
- 98. 98
- 99. 99
- 100. 100 governance Changes in producers might impact consumers
- 102. 102
- 103. 103
- 105. 105 105 Let bad citizens wander around
- 106. 106
- 107. 107 107 Leverage Security, ACL and Quota Security Authorization and ACLs Enforcing Client Quotas
- 108. 108 108 Installing prod on Sunday night
- 109. 109
- 110. 110 configuration If you use the default configuration… You will have issues!
- 111. 111 111 Please read the doc Running Kafka in Production Running ZooKeeper in Production
- 112. 112 112 Not configuring your OS
- 113. 113
- 114. 114 os Tune at least your open file descriptors and mmap count.
- 115. 115 115 Configure your os Running Kafka in Production
- 123. 123 123 Call external services in Kafka Streams
- 124. 124 124 Questions
Editor's Notes
- #5: This is how a company is built Event-driven architecture isn’t new, but different enough now that it is a new type of animal. If we’re successful this will be a major data platform in companies and will redefine the architecture of a digital company. The people here will be part of making that happen.
- #7: So what is a streaming platform? There are a set of core capabilities around data streams you have to have... The first is the ability to publish and subscribe to streams of data. This is something that’s been around for a long time. Messaging systems have been able to do this. What’s different now is the ability to store data and do it properly in a replicated manner. The final capability is to be able to process these streams of data. Initially starting off as a messaging system, over the years Kafka has evolved into a full-fledged distributed streaming platform that embodies the quintessential characteristics of this new category of infrastructure .