Kafka Streams for Java enthusiasts

Chicago Java Users Group
&
Chicago Advanced Analytics Meetup
June 8th 2017
Slim Baltagi
Kafka Streams For Java Enthusiasts

Agenda
1. Apache Kafka: a Streaming Data Platform
2. Overview of Kafka Streams
3. Writing, deploying and running your first
Kafka Streams application
4. Code and Demo of an end-to-end Kafka-
based Streaming Data Application
5. Where to go from here for further learning?
2

Most of what a business does can be thought as event
streams. They are in a
• Retail system: orders, shipments, returns, …
• Financial system: stock ticks, orders, …
• Web site: page views, clicks, searches, …
• IoT: sensor readings, …
and so on.
4

Apache Kafka is an open source streaming data platform (a new
category of software!) with 3 major components:
1. Kafka Core: A central hub to transport and store event streams in
real-time.
2. Kafka Connect: A framework to import event streams from other
source data systems into Kafka and export event streams from
Kafka to destination data systems.
3. Kafka Streams: A Java library to process event streams live as
they occur.
5

Unix Pipelines Analogy
$ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt
Kafka Core:
Unix pipes
Kafka Connect:
I/O redirection
Kafka Streams:
Unix commands
• Kafka Core: is the distributed, durable equivalent of Unix pipes. Use it to connect and
compose your large-scale data applications.
• Kafka Streams are the commands of your Unix pipelines. Use it to transform data
stored in Kafka.
• Kafka Connect is the I/O redirection in your Unix pipelines. Use it to get your data into
and out of Kafka.

2. Overview of Kafka Streams
2.1 Before Kafka Streams?
2.2 What is Kafka Streams?
2.3 Why Kafka Streams?
2.4 What are Kafka Streams key concepts?
2.5 Kafka Streams APIs and code
examples?
7

2.1 Before Kafka Streams?
Before Kafka Streams, to process the data in Kafka you
have 4 options:
• Option 1: Dot It Yourself (DIY) – Write your own
‘stream processor’ using Kafka client libs, typically with
a narrower focus.
• Option 2: Use a library such as AkkaStreams-Kafka,
also known as Reactive Kafka, RxJava, or Vert.x
• Option 3: Use an existing open source stream
processing framework such as Apache Storm, Spark
Streaming, Apache Flink or Apache Samza for
transforming and combining data streams which live in
Kafka…
• Option 4: Use an existing commercial tool for stream
processing with adapter to Kafka such as IBM
InfoSphere Streams, TIBCO StreamBase, …
Each one of the 4 options above of processing data in Kafka
has advantages and disadvantages. 8

2.2 What is Kafka Streams?
 Available since Apache Kafka 0.10 release in May
2016, Kafka Streams is a lightweight open source
Java library for building stream processing applications
on top of Kafka.
 Kafka Streams is designed to consume from & produce
data to Kafka topics.
 It provides a Low-level API for building topologies of
processors, streams and tables.
 It provides a High-Level API for common patterns like
filter, map, aggregations, joins, stateful and stateless
processing.
 Kafka Streams inherits operational characteristics (
low latency, elasticity, fault-tolerance, …) from Kafka.
 A library is simpler than a framework and is easy to
integrate with your existing applications and services!
 Kafka Streams runs in your application code and
imposes no change in the Kafka cluster infrastructure, or
within Kafka.
9

What is Kafka Streams? Java analogy
10

2.3 Why Kafka Streams?
 Processing data in Kafka with Kafka Streams has
the following advantages:
• No need to run another framework or tool for
stream processing as Kafka Streams is already
a library included in Kafka
• No need of external infrastructure beyond
Kafka. Kafka is already your cluster!
• Operational simplicity obtained by getting rid
of an additional stream processing cluster
• As a normal library, it is easier to integrate with
your existing applications and services
• Inherits Kafka features such as fault-
tolerance, scalability, elasticity, authentication,
authorization
• Low barrier to entry: You can quickly write and
run a small-scale proof-of-concept on a single
machine
11

2.4 Wat are Kafka Streams key concepts?
 KStream and KTable as the two basic abstractions.
The distinction between them comes from how the key-
value pairs are interpreted:
• In a stream, each key-value is an independent
piece of information. For example, in a stream of
user addresses: Alice -> New York, Bob -> San
Francisco, Alice -> Chicago, we know that Alice lived
in both cities: New York and Chicago.
• If the table contains a key-value pair for the same
key twice, the latter overwrites the mapping. For
example, a table of user addresses with Alice ->
New York, Bob -> San Francisco, Alice ->
Chicago means that Alice moved from New York to
Chicago, not that she lives at both places at the
same time.
 There’s a duality between the two concepts: a
stream can be viewed as a table, and a table as a
stream. See more on this in the documentation:
http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables 12

KStream vs KTable
record
stream
changelog
stream
When you need…
so that the topic is
interpreted as a
All the values of a key
Latest value of a key
KStream
KTable
then you’d read
the
Kafka topic into aExample
All the cities Alice
has ever lived in
In what city Alice
lives right now?
with
messages
interpreted as
INSERT
(append)
UPDATE
(overwrite existing)
KStream = immutable log
KTable = mutable materialized view

 Event Time: A critical aspect in stream processing is the
notion of time, and how it is modeled and integrated.
• Event time: The point in time when an event or data record
occurred, i.e. was originally created “by the source”.
• Ingestion time: The point in time when an event or data
record is stored in a topic partition by a Kafka broker.
• Processing time: The point in time when the event or data
record happens to be processed by the stream processing
application, i.e. when the record is being consumed.
14

Kafka
Streams
App
App
App
App
1 Capture business
events in Kafka
2 Process the events
with Kafka Streams
4 Other apps query external
systems for latest results
! Must use external systems
to share latest results
App
App
App
1 Capture business
events in Kafka
2 Process the events
with Kafka Streams
3 Now other apps can directly
query the latest results
Before (0.10.0)
After (0.10.1): simplified, more app-centric architecture
Kafka
Streams
App
Interactive Queries: Local queryable state
See blogs:
• Why local state is a fundamental primitive in stream processing? Jay Kreps, July 31st 2014
https://www.oreilly.com/ideas/why-local-state-is-a-fundamental-primitive-in-stream-processing
• Unifying Stream Processing and Interactive Queries in Apache Kafka, Eno Thereska,
October 26th 2016 https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/

 Windowing: Windowing lets you control how
to group records that have the same key for stateful
operations such as aggregations or joins into so-
called windows.
 More concepts in Kafka Streams documentation:
http://docs.confluent.io/current/streams/concepts.htm 16

API option 1: DSL (high level, declarative)
KStream<Integer, Integer> input =
builder.stream("numbers-topic");
// Stateless computation
KStream<Integer, Integer> doubled =
input.mapValues(v -> v * 2);
// Stateful computation
KTable<Integer, Integer> sumOfOdds = input
.filter((k,v) -> v % 2 != 0)
.selectKey((k, v) -> 1)
.groupByKey()
.reduce((v1, v2) -> v1 + v2, "sum-of-odds");
The preferred API for most use cases.
The DSL particularly appeals to users:
• familiar with Spark, Flink, Beam
• fans of Scala or functional
programming
2.5 Kafka Streams APIs and code examples?
• If you’re used to the functions that real-time processing systems like Apache
Spark, Apache Flink, or Apache Beam expose, you’ll be right at home in the
DSL.
• If you’re not, you’ll need to spend some time understanding what methods
like map, flatMap, or mapValues mean. 17

Code Example 1: complete app using DSL
Word
Count
App
configuration
Define
processing
(here:
WordCount)
Start processing
18

API option 2: Processor API (low level, imperative)
class PrintToConsoleProcessor
implements Processor<K, V> {
@Override
public void init(ProcessorContext context) {}
@Override
void process(K key, V value) {
System.out.println("Got value " + value);
}
@Override
void punctuate(long timestamp) {}
@Override
void close() {}
}
Full flexibility but more manual work:
 The Processor API appeals to users:
• familiar with Storm, Samza
• Still, check out the DSL!
• requiring functionality that is
not yet available in the DSL
 Some people have begun using the
low-level Processor API to port their
Apache Storm code to Kafka
Streams.
19

Code Example 2: Complete app using Processor API
20

3. Writing, deploying and running
your first Kafka Streams application
• Step 1: Ensure Kafka cluster is
accessible and has data to process
• Step 2: Write the application code in Java
or Scala
• Step 3: Packaging and deploying the
application
• Step 4: Run the application
21

Step 1: Ensure Kafka cluster is accessible and has
data to process
Get the input data into Kafka via:
• Kafka Connect (part of Apache Kafka)
• or your own application that write data into Kafka
• or tools such as StreamSets, Apache Nifi, ...
Kafka Streams will then be used to process the data
and write the results back to Kafka.
22

Step 2: Write the application code in Java or Scala
• How to start?
• Learn from existing code examples:
https://github.com/confluentinc/examples
• Documentation: http://docs.confluent.io/current/streams/
• How do I install Kafka Streams?
• There is no “installation”! It’s a Java library. Add
it to your client applications like any other Java
library.
• Example adding ‘kafka-streams’ library using
Maven:
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.2.0</version>
</dependency>
23

Step 3: Packaging and deploying the application
How do you package and deploy your Kafka Streams
apps?
• Whatever works for you! Stick to what you/your
company think is the best way for deploying and
packaging a java application.
• Kafka Streams integrates well with what you already
use because an application that uses Kafka Streams
is a normal Java application.
24

Step 4: Run the application
• You don’t need to install a cluster as in other stream
processors (Storm, Spark Streaming, Flink, …) and
submit jobs to it!
• Kafka Streams runs as part of your client
applications, it does not run in the Kafka brokers.
• In production, bundle as fat jar, then `java -cp my-
fatjar.jar com.example.MyStreamsApp`
http://docs.confluent.io/current/streams/developer-guide.html#running-a-kafka-
streams-application
• TIP: During development from your IDE or from CLI, the
‘Kafka Streams Application Reset Tool’, available
since Apache Kafka 0.10.0.1, is great for playing
around.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Application+Res
et+Tool
25

Example: complete app, ready for production at
large-scale!
26

4. Code and Demo of an end-to-end Kafka-
based Streaming Data Application
4.1 Scenario of this demo
4.2 Architecture of this demo
4.3 Setup of this demo
4.4 Results of this demo
4.5 Stopping the demo!

4.1. Scenario of this demo
This demo consists of:
• reading live stream of data (tweets) from Twitter
using Kafka Connect connector for Twitter
• storing them in Kafka broker leveraging Kafka Core
as publish-subscribe message system.
• performing some basic stream processing on tweets
in Avro format from a Kafka topic using Kafka
Streams library to do the following:
• Raw word count - every occurrence of individual words is
counted and written to the topic wordcount (a predefined
list of stopwords will be ignored)
• 5-Minute word count - words are counted per 5 minute
window and every word that has more than 3 occurrences is
written to the topic wordcount5m
• Buzzwords - a list of special interest words can be defined
and those will be tracked in the topic buzzwords
28

4.1. Scenario of this demo
This demo is adapted from one that was given by
Sönke Liebau on July 27th 2016 from OpenCore,
Germany. See blog entry titled: ‘Processing Twitter
Data with Kafka Streams”
http://www.opencore.com/blog/2016/7/kafka-streams-demo/ and
related code at GitHub
https://github.com/opencore/kafkastreamsdemo
In addition:
• I’m using a Docker container instead of the
confluent platform they are providing with their
Virtual Machine defined in Vagrant.
• I’m also using Kafka Connect UI from Landoop for
easy and fast configuration of Twitter connector and
also other Landoop’s Fast Data Web UIs.
29

4.2. Architecture of this demo
30

4.3. Setup of this demo
Step 1: Setup your Kafka Development Environment
Step 2: Get twitter credentials to connect to live data
Step 3: Get twitter live data into Kafka broker
Step 4: Write and test the application code in Java
31

The easiest way to get up and running quickly is to use a Docker container with all
components needed.
First, install Docker on your desktop or on the cloud
https://www.docker.com/products/overview and start it
32
32

Second, install Fast-data-dev, a Docker image for Kafka developers which is
packaging:
• Kafka broker
• Zookeeper
• Open source version of the Confluent Platform with its Schema registry, REST
Proxy and bundled connectors
• Certified DataMountaineer Connectors (ElasticSearch, Cassandra, Redis, ..)
• Landoop's Fast Data Web UIs : schema-registry, kafka-topics, kafka-connect.
• Please note that Fast Data Web UIs are licensed under BSL. You should contact
Landoop if you plan to use them on production clusters with more than 4 nodes.
by executing the command below, while Docker is running and you are connected
to the internet:
docker run --rm -it --net=host landoop/fast-data-dev
• If you are on Mac OS X, you have to expose the ports instead:
docker run --rm -it
-p 2181:2181 -p 3030:3030 -p 8081:8081
-p 8082:8082 -p 8083:8083 -p 9092:9092
-e ADV_HOST=127.0.0.1
landoop/fast-data-dev
• This will download the fast-data-dev Docker image from the Dock Hub.
https://hub.docker.com/r/landoop/fast-data-dev/
• Future runs will use your local copy.
• More details about Fast-data-dev docker image https://github.com/Landoop/fast-data-dev
33

Points of interest:
• the -p flag is used to publish a network port. Inside the
container, ZooKeeper listens at 2181 and Kafka at 9092. If
we don’t publish them with -p, they are not available
outside the container, so we can’t really use them.
• the –e flag sets up environment variables.
• the last part specifies the image we want to run:
landoop/fast-data-dev
• Docker will realize it doesn’t have the landoop/fast-data-
dev image locally, so it will first download it.
That's it.
• Your Kafka Broker is at localhost:9092,
• your Kafka REST Proxy at localhost:8082,
• your Schema Registry at localhost:8081,
• your Connect Distributed at localhost:8083,
• your ZooKeeper at localhost:2181
34

At http://localhost:3030, you will find Landoop's Web UIs for:
• Kafka Topics
• Schema Registry
• as well as a integration test report for connectors & infrastructure
using Coyote. https://github.com/Landoop/coyote
If you want to stop all services and remove everything, simply
hit Control+C.
35

Step 1: Setup your kafka Development Environment
Explore Integration test results at http://localhost:3030/coyote-tests/
36

Step 2: Get twitter credentials to connect to live data
Now that our single-node Kafka cluster is fully up and
running, we can proceed to preparing the input data:
• First you need to register an application with Twitter.
• Second, once the application is created copy the Consumer key and
Consumer Secret.
• Third, generate the Access Token Access and Secret Token required to give
your twitter account access to the new application
Full instructions are here: https://apps.twitter.com/app/new
37

First, create a new Kafka Connect for Twitter
38

Second, configure this Kafka Connect for Twitter to write to the
topic twitter by entering your own track.terms and also the values
of twitter.token, twitter.secret, twitter.comsumerkey and
twitter.consumer.secret
39

Kafka Connect for Twitter is now configured to write data to
the topic twitter.
40

Data is now being written to the topic twitter.
41

 Instead of writing our own code for this demo, we will be leveraging an existing
code from GitHub by Sonke Liebau:
https://github.com/opencore/kafkastreamsdemo
42

 git clone https://github.com/opencore/kafkastreamsdemo
 Edit the buzzwords.txt file with your own works and probably one of
the twitter terms that you are watching live:
43

 The next step is to run the Kafka Streams application that
processes twitter data.
 First, install Maven http://maven.apache.org/install.html
 Then, compile the code into a fat jar with Maven.
$ mvn package
44

Two jar files will be created in the target folder:
1. KafkaStreamsDemo-1.0-SNAPSHOT.jar – Only your project classes
2. KafkaStreamsDemo-1.0-SNAPSHOT-jar-with-dependencies.jar –
Project and dependency classes in a single jar.
45

 Then
java -cp target/KafkaStreamsDemo-1.0-SNAPSHOT-
jar-with-dependencies.jar
com.opencore.sapwebinarseries.KafkaStreamsDemo
 TIP: During development: from your IDE, from CLI …
Kafka Streams Application Reset Tool, available
since Apache Kafka 0.10.0.1, is great for playing
around.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Application+Reset+Tool
46

4.4. Results of this demo
Once the above is running, the following topics will be
populated with data :
• Raw word count - Every occurrence of individual
words is counted and written to the
topic wordcount (a predefined list of stopwords will
be ignored)
• 5-Minute word count - Words are counted per 5
minute window and every word that has more than
three occurrences is written to the
topic wordcount5m
• Buzzwords - a list of special interest words can be
defined and those will be tracked in the
topic buzzwords - the list of these words can be
defined in the file buzzwords.txt
47

4.4. Results of this demo
Accessing the data generated by the code is as
simple as starting a console consumer which is shipped
with Kafka
• You need first to enter the container to use any tool as you like:
docker run --rm -it --net=host landoop/fast-data-dev bash
• Use the following command to check the topics:
• kafka-console-consumer --topic wordcount --new-
consumer --bootstrap-server 127.0.0.1:9092 --property
print.key=true
• kafka-console-consumer --topic wordcount5m --new-
print.key=true
• kafka-console-consumer --topic buzzwords --new-
print.key=true
48

4.5. Stopping the demo!
To stop the Kafka Streams Demo application:
• $ ps – A | grep java
• $ kill -9 PID
If you want to stop all services in fast-data-dev Docker
image and remove everything, simply hit Control+C.
50

 Kafka Streams code examples
• Apache Kafka
https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apac
he/kafka/streams/examples
• Confluent https://github.com/confluentinc/examples/tree/master/kafka-streams
 Source Code https://github.com/apache/kafka/tree/trunk/streams
 Kafka Streams Java docs
http://docs.confluent.io/current/streams/javadocs/index.html
 First book on Kafka Streams (MEAP)
• Kafka Streams in Action https://www.manning.com/books/kafka-streams-
in-action
 Kafka Streams download
• Apache Kafka https://kafka.apache.org/downloads
• Confluent Platform http://www.confluent.io/download
51

 Kafka Users mailing list https://kafka.apache.org/contact
 Kafka Streams at Confluent Community on Slack
• https://confluentcommunity.slack.com/messages/streams/
 Free ebook:
• Making Sense of Stream processing by Martin
Klepmann https://www.confluent.io/making-sense-of-stream-processing-
ebook-download/
 Kafka Streams documentation
• Apache Kafka http://kafka.apache.org/documentation/streams
• Confluent http://docs.confluent.io/3.2.0/streams/
 All web resources related to Kafka Streams
http://sparkbigdata.com/component/tags/tag/69-kafka-streams 52

Thank you!
Let’s keep in touch!
@SlimBaltagi
https://www.linkedin.com/in/slimbaltagi
sbaltagi@gmail.com
53

Kafka Streams for Java enthusiasts

Recommended

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Kafka Streams for Java enthusiasts (20)

More from Slim Baltagi (13)

Recently uploaded (20)

Kafka Streams for Java enthusiasts