躬行系列-hadoop集群部署

Java安装

[root@localhost jdk-11.0.15]# cd /etc/profile.d/
[root@localhost profile.d]# vim my_env.sh
[root@localhost profile.d]# source /etc/profile

机器初始配置

因为我这是虚拟机,所以要配置固定ip。如果你也和我一样是自己测试，一定注意netmask和gateway要更换成自己的。

vim /etc/sysconfig/network-scripts/ifcfg-enp0s3
------------------------------------- 
IPADDR="192.168.0.114"
NETMASK="255.255.255.0"
GATEWAY="192.168.0.1"
DNS1="114.114.114.114"

hostname

hostnamectl set-hostname hadoop-1

cat >> /etc/hosts << EOF
192.168.0.114 hadoop-1
192.168.0.115 hadoop-2
192.168.0.116 hadoop-3
EOF

集群规划

ip	name	组件
192.168.0.114	hadoop-1	namenode datanode journalnode
192.168.0.115	hadoop-2	namenode datanode journalnode
192.168.0.116	hadoop-3	namenode datanode journalnode

配置

上传解压等忽略
两个核心配置文件:core-site.xml hdfs-site.xml
位置：/usr/hadoop/hadoop-3.3.3/etc/hadoop

环境变量

配置自己的环境变量地址,无需配置在原有的系统profile中
/etc/profile.d/my_env.sh
启动的话只需要
source /etc/profile.d/my_env.sh

export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

source /etc/profile

hadoop version

常用命令

启动journalnode
hdfs --daemon start journalnode
启动namenode
hdfs namenode -format
hdfs --daemon start namenode
hdfs namenode -bootstrapStandby
zookeeper启动
./zkServer.sh start
zkfc初始化
hdfs zkfc -formatZK
集群一键启停
start-dfs.sh
stop-dfs.sh
start-yarn.sh
stop-yarn.sh
获取节点状态
hdfs haadmin -getServiceState nn2
手动切换节点状态为active
hdfs haadmin -transitionToActive --forcemanual nn2
hadoop手动上传文件
hadoop fs -put README.md /
获取yarn节点状态
yarn rmadmin -getServiceState rm1
查询zookeeper集群状态
./zkServer.sh status
启停history server
./mapred --daemon start historyserver

web控制台

hadoop:http://192.168.0.116:9870/
yarn:http://hadoop-1:8088/
history server:http://hadoop-1:19888/jobhistory

Hadoop性能测试

写测试
hadoop jar /usr/local/hadoop/hadoop-3.3.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.3-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB
读测试
hadoop jar /usr/local/hadoop/hadoop-3.3.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.3-tests.jar TestDFSIO -read -nrFiles 5 -fileSize 128MB
mapreduce测试
至少要10台服务器+1T*128G才能跑

启动kafka

bin/kafka-server-start.sh -daemon /usr/local/kafka/kafka_2.13-3.2.1/config/server.properties