Hadoop集群部署与启动，Yarn模式要考虑Container内存资源分配

Hadoop集群的部署

开始安装Hadoop之前，为了让 Master 节点可以无密码 SSH 登陆到各个 Slave 节点上，所以需要配置 SSH 无密码登陆

安装版本： hadoop-2.8.3.tar.gz

mkdir /usr/local/hadoop
tar zxvf hadoop-2.8.3.tar.gz -C /usr/local/hadoop

修改域名与IP的对应关系(hadoop2和hadoop3同样也需要修改hosts文件）

vi /etc/hosts
10.2.15.176 hadoop1
10.2.15.177 hadoop2
10.2.15.170 hadoop3

配置环境变量(hadoop2和hadoop3同样也需要修改hosts文件）

vi /etc/profile
export FLINK_HOME=/usr/local/hadoop/hadoop-2.8.3
export PATH=$FLINK_HOME/bin:$PATH
source /etc/profile

先建好稍后需要用到的文件夹

mkdir /usr/local/hadoop
mkdir /usr/local/hadoop/tmp
mkdir /usr/local/hadoop/var
mkdir /usr/local/hadoop/dfs
mkdir /usr/local/hadoop/dfs/name
mkdir /usr/local/hadoop/dfs/data

修改core-site.xml文件

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:9000</value>
</property>
</configuration>

修改mapred-site.xml文件

cp /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-site.xml  
vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-site.xml  
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1:19888</value>
</property>

修改hdfs-site.xml文件

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>2</value>
     <description>HDFS 的数据块的副本存储个数, 默认是3</description>
  </property>
  <property>
      <name>dfs.permissions</name>
      <value>false</value>
      <description>need not permissions</description>
</property>
</configuration>

修改hdfs-site.xml文件

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1:8088</value>
</property>
<property>
  <name>yarn.resourcemanager.am.max-attempts</name>
  <value>4</value>
  <description>
    The maximum number of application master execution attempts.
  </description>
</property>

如果是以Flink on Yarn方式启动的，因为Hadoop Yarn是一个资源调度器，所以我们应该考虑好每个Conatiner被分配到的内存资源，所以需要在文件hdfs-site.xml中配置好 yarn.nodemanager.resource.memory-mb, yarn.scheduler.minimum-allocation-mb, yarn.scheduler.maximum-allocation-mb, yarn.app.mapreduce.am.resource.mb和yarn.app.mapreduce.am.command-opts，不然会发生内存不足，导致Application启动失败。

Current usage: 303.2 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/yarn-site.xml

<property>
  <name>yarn.nodemanager.vmem-check-enbaled</name>
  <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>106496</value>
</property>
<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>106496</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>4096</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx3276m</value>
</property>

修改 hadoop-env.sh , mapred-env.sh 和 yarn-env.sh

vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/hadoop-env.sh
export JAVA_HOME="/usr/local/jdk/jdk1.8.0_251"
vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/mapred-env.sh
export JAVA_HOME="/usr/local/jdk/jdk1.8.0_251"
vi /usr/local/hadoop/hadoop-2.8.3/etc/hadoop/yarn-env.sh
export JAVA_HOME="/usr/local/jdk/jdk1.8.0_251"

把/hadoop发送给另外两台服务器

scp -r /usr/local/hadoop hadoop2:/usr/local
scp -r /usr/local/hadoop hadoop3:/usr/local

启动Hadoop集群

初始化HDFS系统

/usr/local/hadoop/hadoop-2.8.3/bin/hdfs namenode -format

开启 NameNode 和 DataNode 守护进程

/usr/local/hadoop/hadoop-2.8.3/sbin/start-all.sh

在浏览器中输入 http://hadoop1:50070，可查看相关信息

运行wordcount demo

bin/hdfs dfs -mkdir /input
bin/hdfs dfs -ls /
bin/hdfs dfs -put /usr/local/hadoop/tmp/input_hadoop_demo_test.txt /input/
bin/hdfs dfs -ls /input/
bin/hadoop jar /usr/local/hadoop/hadoop-2.8.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /input/input_hadoop_demo_test.txt /output
bin/hdfs dfs -ls /output
bin/hdfs dfs -cat /output/part-r-00000