KaFka快速搭建单机版

JDK安装和Hadoop安装参考地址:

https://blog.csdn.net/su_mingyang/article/details/120838047?spm=1001.2014.3001.5501

1、安装Kafka集群

1.1、解压kafka

tar -xzvf /chinaskills/kafka_2.11-2.0.0.tgz -C /usr/local/src/	

1.2、重命名为kafka

mv /usr/local/src/kafka_2.11-2.0.0 /usr/local/src/kafka

1.3、配置环境变量(当前用户生效)

vi /root/.bash_profile 

配置内容:

export KAFKA_HOME=/usr/local/src/kafka
export PATH=$PATH:$KAFKA_HOME/bin

1.4、加载环境变量

source  /root/.bash_profile

1.5、启动kafaka-zookeeper

zookeeper-server-start.sh -daemon /usr/local/src/kafka/config/zookeeper.properties

1.6、启动kafak服务

kafka-server-start.sh -daemon /usr/local/src/kafka/config/server.properties

1.7 查看jps进程

[root@master ~]# jps
38304 QuorumPeerMain
38662 Jps
38600 Kafka

2、测试kafka

2.1、创建topic

kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

输出信息:

Created topic "test".

2.2 、生产数据

查看topic

kafka-topics.sh --list --zookeeper localhost:2181

输出信息:

test

运行生产者

kafka-console-producer.sh --broker-list localhost:9092 --topic test

写入数据:

>hello word 
>hello word
>hive kafka flume

2.3 、新开一个终端进行消费数据

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

输出信息:

hello word 
hello word
hive kafka flume

3、配置代理

3.1、复制server.properties文件

cp /usr/local/src/kafka/config/server.properties /usr/local/src/kafka/config/server-1.properties 
cp /usr/local/src/kafka/config/server.properties /usr/local/src/kafka/config/server-2.properties 

3.2、配置代理服务

**server-1.properties **

vi /usr/local/src/kafka/config/server-1.properties 

配置内容

broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1

**server-2.properties **

vi /usr/local/src/kafka/config/server-2.properties 

配置内容

broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2

3.3、 启动代理集群

kafka-server-start.sh -daemon /usr/local/src/kafka/config/server-1.properties 
kafka-server-start.sh -daemon /usr/local/src/kafka/config/server-2.properties 

3.4、查看jps进程

[root@master ~]# jps
38304 QuorumPeerMain
40992 Jps
40931 Kafka
39764 ConsoleProducer
38600 Kafka
40605 Kafka

3.5、创建一个3分区的主题

kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

输出信息:

Created topic "my-replicated-topic".

3.6、查看描述主题

kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

输出信息:

Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2