kafka_2.10-0.10.0.1 集群安装配置使用独立zookeeper集群

zookeeper   kafka   topic  

一、集群环境

  • CentOS 6.6 X64
  • zookeeper集群配置
  • kafka_2.10-0.10.0.1.tgz
  • 集群配置如下,之前已经部署过hadoop和zookeeper
master.hdp.imdst.com  zookeeper.0 kafka.0  
1.slave.hdp.imdst.com zookeeper.1 kafka.1  
2.slave.hdp.imdst.com zookeeper.2 kafka.2  

二、安装kafka

  • 下载、解压
wget http://mirrors.cnnic.cn/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz  
tar zxf kafka_2.10-0.10.0.1.tgz -C /home/hadoop/src  
chown hadoop.hadoop /home/hadoop -R  
  • 配置master
su hadoop  
cd ~/src && ln -s kafka_2.10-0.10.0.1 kafka  
vim kafka/config/server.properties  
------------------------------------
broker.id=0  
listeners=PLAINTEXT://:9092  
advertised.listeners=PLAINTEXT://:9092  
log.dirs=/data/kafka-logs  
zookeeper.connect=master.hdp.imdst.com:2181,1.slave.hdp.imdst.com:2181,2.slave.hdp.imdst.com:2181  
------------------------------------
  • 配置slave 1和2
cd ~/src  
scp -r kafka 1.slave.hdp.imdst.com:/home/hadoop/src/  
scp -r kafka 2.slave.hdp.imdst.com:/home/hadoop/src/  
#分别修改server.properties
broker.id=1 和broker.id=2  

三、启动和关闭

  • 启动
cd ~/src/kafka/bin  
 ./kafka-server-start.sh -daemon  ../config/server.properties 
  • 关闭
cd ~/src/kafka/bin  
./kafka-server-stop.sh

四、在任意一台机器上测试

  • 创建一个名为"mytest-topic"的topic,该topic切分为4份,每一份备份数为3
bin/kafka-topics.sh --create --zookeeper master.hdp.imdst.com:2181 --replication-factor 3 --partitions 4 --topic mytest-topic  
  • 列出所有topic
./kafka-topics.sh --list --zookeeper master.hdp.imdst.com:2181,1.slave.hdp.imdst.com:2181,2.slave.hdp.imdst.com:2181
  • 在zookeeper中可以查看到自己创建的topic
cd ~/src/zookeeper/bin  
./zkCli.sh
Connecting to localhost:2181  
[zk: localhost:2181(CONNECTED) 0] ls /brokers/topics
[mytest-topic, leoiceo-topic]