4

相关环境

软件 版本
Centos 3.10.0-862.el7.x86_64
jdk 1.8
zookeeper 3.4.10
kafka 1.1.0
flume 1.6.0
Host IP
c1 192.168.1.200
c1_1 192.168.1.201
c1_2 192.168.1.202

用户统一为hadoop

前置操作

各主机间启动ssh连接

这一步至关重要,如果没有配置成功,会影响到hadoop,kafka集群之间的连接

[hadoop@c1 ~]$ ssh-keygen
[hadoop@c1 ~]$ sudo vim /etc/ssh/sshd_config
    ...
    PubkeyAuthentication yes
    ...
[hadoop@c1 ~]$ systemctl restart sshd
[hadoop@c1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  # ssh 本机
[hadoop@c1 ~]$ sudo vim /etc/hosts  # 添加三台主机对应ip host
    ...
    192.168.1.200 c1
    192.168.1.201 c1_1
    192.168.1.202 c1_2
    ...
[hadoop@c1 ~]$ ssh-copy-id -i  ~/.ssh/id_rsa.pub hadoop@c1_1
[hadoop@c1 ~]$ ssh-copy-id -i  ~/.ssh/id_rsa.pub hadoop@c1_2

其他两台机器重复上面的操作即可.完成后,可以ssh一下各台机子(包括本机)是否还需要密码

安装软件

# 下载jdk1.8+
[hadoop@c1 ~] mkdir -p ~/app/jdk1.8 && tar -zxvf jdk-8u171-linux-x64.tar.gz -C ~/app/jdk1.8
# 下载flume1.6
[hadoop@c1 ~] tar -zxvf apache-flume-1.6.0-bin.tar.gz -C ~/app/
# 下载zookeeper3.4.10
[hadoop@c1 ~] tar -zxvf zookeeper-3.4.10.tar.gz -C ~/app/
# 下载kafka_2.11-1.1.0
[hadoop@c1 ~] tar -xzf kafka_2.11-1.1.0.tgz -C ~/app/
# 环境变量
[hadoop@c1 ~] vim .bash_profile
    ...
    export JAVA_HOME=/home/hadoop/app/jdk1.8.0
    export PATH=$JAVA_HOME/bin:$PATH
    export FLUME_HOME=/home/hadoop/app/flume-1.6.0
    export PATH=$FLUME_HOME/bin:$PATH
    export ZK_HOME=/home/hadoop/app/zookeeper-3.4.10
    export PATH=$ZK_HOME/bin:$PATH
    export KAFKA_HOME=/home/hadoop/app/kafka_2.11-1.1.0
    export PATH=$KAFKA_HOME/bin:$PATH
    ...
[hadoop@c1 ~] source .bash_profile
# 复制软件和环境变量到其他主机
[hadoop@c1 ~]scp -r ~/app hadoop@c1_1:~
[hadoop@c1 ~]scp -r ~/app hadoop@c1_2:~
[hadoop@c1 ~]scp .bash_profile hadoop@c1_1:~
[hadoop@c1 ~]scp .bash_profile hadoop@c1_2:~
# 到其他主机执行source .bash_profile

配置文件

  • flume配置文件

    # vim ${FLUME_HOME}/conf/nginx_kafka.conf
    nginx-kafka.sources = r1                                                                                     
    nginx-kafka.sinks = k1
    nginx-kafka.channels = c1
    
    nginx-kafka.sources.r1.type = exec
    nginx-kafka.sources.r1.command = tail -f /home/hadoop/data/access.log
    nginx-kafka.sources.r1.shell = /bin/sh -c
    # flume1.6 kafka sink 写法
    nginx-kafka.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
    nginx-kafka.sinks.k1.brokerList = c1:9092
    nginx-kafka.sinks.k1.topic= nginxtopic
    nginx-kafka.sinks.k1.batchSize=10
    
    nginx-kafka.channels.c1.type = memory
    nginx-kafka.sources.r1.channels = c1
    nginx-kafka.sinks.k1.channel = c1
    
  • zookeeper配置文件

    # cp ${ZK_HOME}/conf/zoo_simple.cfg ${ZK_HOME}/conf/zoo.cfg && vim ${ZK_HOME}/conf/zoo.cfg
    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/home/hadoop/data/zookeeper
    clientPort=2181
    # 注意,当前主机的zookeeper server不能设置hostname,必须是0.0.0.0 否则无法连接
    server.1=0.0.0.0:2888:3888                                                   
    server.2=c1_1:2888:3888
    server.3=c1_2:2888:3888

    创建zookeeper集群id

    echo "1">/home/hadoop/data/zookeeper/myid

    其他主机重复相同操作,server.x需要和myid值一致,

  • kafka配置文件

    kafka配置文件需要改动的只有几个

    # ${KAFKA_HOME}/config/server.properties
    broker.id=0
    host.name=c1
    listeners=PLAINTEXT://192.168.1.200:9092
    advertised.listeners=PLAINTEXT://c1:9092
    zookeeper.connect=c1:2181,c1_1:2181,c1_2:2181

    broker.id从0开始且在集群中唯一

    listeners需要填上IP
    advertised.listeners需要填上hostname

    这里我这么设置是没问题的,但是不清楚为啥这么设置

    其他主机kafka配置文件同样的操作

编写集群启动脚本

  • zookeeper集群脚本

    # vim start_zookeeper.sh
    #!/bin/bash
    echo "start zkServer..."
    for i in c1 c1_1 c1_2
        do
            ssh hadoop@$i "source ~/.bash_profile;zkServer.sh start"
        done   
    # vim stop_zookeeper.sh
    #!/bin/bash
    echo "stop zkServer..."
    for i in c1 c1_1 c1_2
        do
            ssh hadoop@$i "source ~/.bash_profile;zkServer.sh stop"             
        done
    

    chmod a+x start_zookeeper.sh stop_zookeeper.sh

  • kafka集群脚本

    # vim start_kafka.sh
    #!/bin/sh
    echo "start kafka..."
    for i in c1 c1_1 c1_2
    do
        ssh hadoop@$i "source ~/.bash_profile;kafka-server-start.sh -daemon ${KAFKA_HOME}/config/server.properties &"
        echo "done"                                                             
    done
    
    # vim stop_kafka.sh
    #!/bin/sh
    echo "stop kafka..."
    for i in c1 c1_1 c1_2
    do
        ssh hadoop@$i "source ~/.bash_profile;kafka-server-stop.sh"            
    done

    chmod a+x start_kafka.sh stop_kafka.sh

实战

  1. 启动程序

    # 启动zookeeper
    [hadoop@c1 ~]$ ./start_zookeeper.sh
    [hadoop@c1 ~]$ zkServer.sh status
    ZooKeeper JMX enabled by default
    Using config: /home/hadoop/zookeeper-3.4.10/bin/../conf/zoo.cfg
    Mode: follower
    # 启动kafka
    [hadoop@c1 ~]$ ./start_kafka.sh
    [hadoop@c1 ~]$ jps
    2953 QuorumPeerMain  # zookeeper 进程
    3291 Kafka  #kafka进程
    3359 Jps
  2. 创建Topic

    [hadoop@c1 ~]$ kafka-topics.sh --create --zookeeper c1:2181,c1_1:2181,c1_2:2181 --replication-factor 3 --partitions 1 --topic nginxtopic
  3. 检查Topic

    [hadoop@c1 ~]$ kafka-topics.sh --zookeeper c1:2181,c1_1:2181,c1_2:2181 --list
    nginx
  4. 启动消费者

    [hadoop@c1 ~]$ kafka-console-consumer.sh --bootstrap-server c1:9092,c1_1:9092,c1_2:9092 -topic nginxtopic --from-beginning
  5. 模拟日志

    #vim create_log.sh
    ---
    #!/bin/sh
    # access.log-xxx 等多个文件是生产环境拖下来的真是日志
    cat access.log-*| while read -r line
    do
    echo $line >> /home/hadoop/logs/access.log
    sleep 0.$(($RANDOM%5+1))  # 防止日志写入过快
    done
  6. 启动flume

    新开一个窗口

    [hadoop@c1 ~]$ flume-ng agent --conf-file  conf/nginx_kafka.conf -c conf/ --name nginx-kafka -Dflume.root.logger=DEBUG,console

稍等片刻后

flume输出日志

图片描述

kafka-console-consume 输出日志

图片描述

至此项目已经完全跑起来了~

错误排查及解决

  • not in the sudoers file. This incident will be reported
    没有sudo的操作权限,需要在root权限下编辑/etc/sudoer

    ...
    ## Allow root to run any commands anywhere 
    root    ALL=(ALL)   ALL
    hadoop  ALL=(ALL)   ALL
    ...
  • 已经添加过ssh仍需要输入密码

    chmod 700 ~/.ssh
    chmod 644 ~/.ssh/authorized_keys
  • zookeeper: It is probably not running

    1. 有可能是ssh无法免密访问其他主机
    2. 有可能是没有正确的写myid

可以在zookeeper.out 查看详细的错误信息


amoyiki
126 声望27 粉丝