相关环境
软件 | 版本 |
---|---|
Centos | 3.10.0-862.el7.x86_64 |
jdk | 1.8 |
zookeeper | 3.4.10 |
kafka | 1.1.0 |
flume | 1.6.0 |
Host | IP |
---|---|
c1 | 192.168.1.200 |
c1_1 | 192.168.1.201 |
c1_2 | 192.168.1.202 |
用户统一为hadoop
前置操作
各主机间启动ssh连接
这一步至关重要,如果没有配置成功,会影响到hadoop,kafka集群之间的连接
[hadoop@c1 ~]$ ssh-keygen
[hadoop@c1 ~]$ sudo vim /etc/ssh/sshd_config
...
PubkeyAuthentication yes
...
[hadoop@c1 ~]$ systemctl restart sshd
[hadoop@c1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys # ssh 本机
[hadoop@c1 ~]$ sudo vim /etc/hosts # 添加三台主机对应ip host
...
192.168.1.200 c1
192.168.1.201 c1_1
192.168.1.202 c1_2
...
[hadoop@c1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@c1_1
[hadoop@c1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@c1_2
其他两台机器重复上面的操作即可.完成后,可以ssh一下各台机子(包括本机)是否还需要密码
安装软件
# 下载jdk1.8+
[hadoop@c1 ~] mkdir -p ~/app/jdk1.8 && tar -zxvf jdk-8u171-linux-x64.tar.gz -C ~/app/jdk1.8
# 下载flume1.6
[hadoop@c1 ~] tar -zxvf apache-flume-1.6.0-bin.tar.gz -C ~/app/
# 下载zookeeper3.4.10
[hadoop@c1 ~] tar -zxvf zookeeper-3.4.10.tar.gz -C ~/app/
# 下载kafka_2.11-1.1.0
[hadoop@c1 ~] tar -xzf kafka_2.11-1.1.0.tgz -C ~/app/
# 环境变量
[hadoop@c1 ~] vim .bash_profile
...
export JAVA_HOME=/home/hadoop/app/jdk1.8.0
export PATH=$JAVA_HOME/bin:$PATH
export FLUME_HOME=/home/hadoop/app/flume-1.6.0
export PATH=$FLUME_HOME/bin:$PATH
export ZK_HOME=/home/hadoop/app/zookeeper-3.4.10
export PATH=$ZK_HOME/bin:$PATH
export KAFKA_HOME=/home/hadoop/app/kafka_2.11-1.1.0
export PATH=$KAFKA_HOME/bin:$PATH
...
[hadoop@c1 ~] source .bash_profile
# 复制软件和环境变量到其他主机
[hadoop@c1 ~]scp -r ~/app hadoop@c1_1:~
[hadoop@c1 ~]scp -r ~/app hadoop@c1_2:~
[hadoop@c1 ~]scp .bash_profile hadoop@c1_1:~
[hadoop@c1 ~]scp .bash_profile hadoop@c1_2:~
# 到其他主机执行source .bash_profile
配置文件
-
flume配置文件
# vim ${FLUME_HOME}/conf/nginx_kafka.conf nginx-kafka.sources = r1 nginx-kafka.sinks = k1 nginx-kafka.channels = c1 nginx-kafka.sources.r1.type = exec nginx-kafka.sources.r1.command = tail -f /home/hadoop/data/access.log nginx-kafka.sources.r1.shell = /bin/sh -c # flume1.6 kafka sink 写法 nginx-kafka.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink nginx-kafka.sinks.k1.brokerList = c1:9092 nginx-kafka.sinks.k1.topic= nginxtopic nginx-kafka.sinks.k1.batchSize=10 nginx-kafka.channels.c1.type = memory nginx-kafka.sources.r1.channels = c1 nginx-kafka.sinks.k1.channel = c1
-
zookeeper配置文件
# cp ${ZK_HOME}/conf/zoo_simple.cfg ${ZK_HOME}/conf/zoo.cfg && vim ${ZK_HOME}/conf/zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/home/hadoop/data/zookeeper clientPort=2181 # 注意,当前主机的zookeeper server不能设置hostname,必须是0.0.0.0 否则无法连接 server.1=0.0.0.0:2888:3888 server.2=c1_1:2888:3888 server.3=c1_2:2888:3888
创建zookeeper集群id
echo "1">/home/hadoop/data/zookeeper/myid
其他主机重复相同操作,
server.x
需要和myid值一致, -
kafka配置文件
kafka配置文件需要改动的只有几个
# ${KAFKA_HOME}/config/server.properties broker.id=0 host.name=c1 listeners=PLAINTEXT://192.168.1.200:9092 advertised.listeners=PLAINTEXT://c1:9092 zookeeper.connect=c1:2181,c1_1:2181,c1_2:2181
broker.id
从0开始且在集群中唯一listeners
需要填上IPadvertised.listeners
需要填上hostname这里我这么设置是没问题的,但是不清楚为啥这么设置
其他主机kafka配置文件同样的操作
编写集群启动脚本
-
zookeeper集群脚本
# vim start_zookeeper.sh #!/bin/bash echo "start zkServer..." for i in c1 c1_1 c1_2 do ssh hadoop@$i "source ~/.bash_profile;zkServer.sh start" done
# vim stop_zookeeper.sh #!/bin/bash echo "stop zkServer..." for i in c1 c1_1 c1_2 do ssh hadoop@$i "source ~/.bash_profile;zkServer.sh stop" done
chmod a+x start_zookeeper.sh stop_zookeeper.sh
-
kafka集群脚本
# vim start_kafka.sh #!/bin/sh echo "start kafka..." for i in c1 c1_1 c1_2 do ssh hadoop@$i "source ~/.bash_profile;kafka-server-start.sh -daemon ${KAFKA_HOME}/config/server.properties &" echo "done" done
# vim stop_kafka.sh #!/bin/sh echo "stop kafka..." for i in c1 c1_1 c1_2 do ssh hadoop@$i "source ~/.bash_profile;kafka-server-stop.sh" done
chmod a+x start_kafka.sh stop_kafka.sh
实战
-
启动程序
# 启动zookeeper [hadoop@c1 ~]$ ./start_zookeeper.sh [hadoop@c1 ~]$ zkServer.sh status ZooKeeper JMX enabled by default Using config: /home/hadoop/zookeeper-3.4.10/bin/../conf/zoo.cfg Mode: follower # 启动kafka [hadoop@c1 ~]$ ./start_kafka.sh [hadoop@c1 ~]$ jps 2953 QuorumPeerMain # zookeeper 进程 3291 Kafka #kafka进程 3359 Jps
-
创建Topic
[hadoop@c1 ~]$ kafka-topics.sh --create --zookeeper c1:2181,c1_1:2181,c1_2:2181 --replication-factor 3 --partitions 1 --topic nginxtopic
-
检查Topic
[hadoop@c1 ~]$ kafka-topics.sh --zookeeper c1:2181,c1_1:2181,c1_2:2181 --list nginx
-
启动消费者
[hadoop@c1 ~]$ kafka-console-consumer.sh --bootstrap-server c1:9092,c1_1:9092,c1_2:9092 -topic nginxtopic --from-beginning
-
模拟日志
#vim create_log.sh --- #!/bin/sh # access.log-xxx 等多个文件是生产环境拖下来的真是日志 cat access.log-*| while read -r line do echo $line >> /home/hadoop/logs/access.log sleep 0.$(($RANDOM%5+1)) # 防止日志写入过快 done
-
启动flume
新开一个窗口
[hadoop@c1 ~]$ flume-ng agent --conf-file conf/nginx_kafka.conf -c conf/ --name nginx-kafka -Dflume.root.logger=DEBUG,console
稍等片刻后
flume输出日志
kafka-console-consume 输出日志
至此项目已经完全跑起来了~
错误排查及解决
-
not in the sudoers file. This incident will be reported
没有sudo的操作权限,需要在root权限下编辑/etc/sudoer
... ## Allow root to run any commands anywhere root ALL=(ALL) ALL hadoop ALL=(ALL) ALL ...
-
已经添加过ssh仍需要输入密码
chmod 700 ~/.ssh chmod 644 ~/.ssh/authorized_keys
-
zookeeper: It is probably not running
- 有可能是ssh无法免密访问其他主机
- 有可能是没有正确的写myid
可以在zookeeper.out
查看详细的错误信息
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用
。你还可以使用@
来通知其他用户。