大数据环境搭建Step2 Hadoop安装配置

集群搭建有三个节点

master 192.168.21.161
slave1 192.168.21.162
slave2 192.168.21.163

软件

Centos7
hadoop-2.7.7

获取软件

Wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

解压安装

mkdir -p /usr/hadoop
tar -xzvf hadoop-2.7.7.tar.gz -C /usr/hadoop/

环境配置

• 环境变量

  /etc/profile文件中添加如下内容

  export HADOOP_HOME=/usr/hadoop/hadoop-2.7.7
  export CLASSPATH=$CLASSPATH:$HADOOP_HOME/lib
  export PATH=$PATH:$HADOOP_HOME/bin

• 生效

  source /etc/profile

Hadoop配置

/usr/hadoop/hadoop-2.7.7/etc/hadoop目录下

• hadoop-env.sh文件添加如下内容

  export JAVA_HOME=/usr/java/jdk1.8.0_171

• core-site.xml文件中添加如下内容

  <configuration>
  <property>
   <name>fs.default.name</name>
    <value>hdfs://master:9000</value>
  </property>
  
  <property>
   <name>hadoop.tmp.dir</name> 
    <value>/usr/hadoop/hadoop-2.7.7/hdfs/tmp</value>
  <description>A base for other temporarydirectories.</description>
  </property>
  
  <property>
   <name>io.file.buffer.size</name> 
    <value>131072</value>
  </property>
  
  <property>
   <name>fs.checkpoint.period</name>
    <value>60</value>
  </property>
  
  <property>
   <name>fs.checkpoint.size</name>
    <value>67108864</value>
  </property>
  </configuration>

• yarn-site.xml文件添加如下内容

  <configuration>
  
  <!-- Site specific YARN configuration properties -->
  
   <property>
   <name>yarn.resourcemanager.address</name>
    <value>master:18040</value>
   </property>
  
   <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:18030</value>
   </property>
  
   <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:18088</value>
   </property>
  
   <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:18025</value>
   </property>
  
   <property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>master:18141</value>
   </property>
  
   <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
   </property>
  
   <property>
   <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
   <!-- Site specific YARNconfiguration properties -->
  </configuration>

• 生成slaves文件，内容如下

  slave1
  slave2

• 生成master文件，内容如下

  master

• 编辑hdfs-site.xml

  <configuration>
   <property>
   <name>dfs.replication</name>
    <value>2</value>
   </property>
  
   <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/usr/hadoop/hadoop-2.7.7/hdfs/name</value>
    <final>true</final>
   </property>
  
   <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/hadoop/hadoop-2.7.7/hdfs/data</value>
    <final>true</final>
   </property>
  
   <property>
   <name>dfs.namenode.secondary.http-address</name>
    <value>master:9001</value>
   </property>
  
   <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
   </property>
  
   <property>
    <name>dfs.permissions</name>
    <value>false</value>
   </property>
  
  </configuration>

• 修改mapred-site.xml文件

  copy模版文件

  cp mapred-site.xml.template mapred-site.xml 

  添加内容

  <configuration>
  <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
   </property>
  </configuration>

  将/usr/hadoop目录scp到slave1和slave2

  scp -r /usr/hadoop root@slave1:/usr/
  scp -r /usr/hadoop root@slave2:/usr/

• slave1和slave2环境变量设置和生效
  /etc/profile文件中添加如下内容

  export HADOOP_HOME=/usr/hadoop/hadoop-2.7.7
  export CLASSPATH=$CLASSPATH:$HADOOP_HOME/lib
  export PATH=$PATH:$HADOOP_HOME/bin

  source /etc/profile

• 格式化hadoop

  在master上执行。slave1和slave2不用执行

  hadoop namenode -format
  /usr/hadoop/hadoop-2.7.7/sbin/start-all.sh

• 确认

  在master，slave1，slave2上执行jps查看执行状况
  master
  

  slave1
  

  slave2
  

  访问master的ip：50070可以看到hdfs的web管理界面