躬行系列-sqoop部署

环境准备

jdk
zookeeper
hadoop
sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
mysql

步骤

• 解压到/usr/local
• 修改配置/usr/local/sqoop-1.4.7.bin__hadoop-2.6.0/conf/sqoop-env-template.sh
• mv sqoop-env-template.sh sqoop-env.sh

• export HADOOP_COMMON_HOME=/usr/local/hadoop-3.3.3
  export HADOOP_MAPRED_HOME=/usr/local/hadoop-3.3.3
  export ZOOKEEPER_HOME=/usr/local/apache-zookeeper-3.7.1-bin
  export ZOOCFGDIR=/usr/local/apache-zookeeper-3.7.1-bin/conf

• 测试连接bin/sqoop list-databases --connect jdbc:mysql://192.168.0.121:3306/ --username root --password root

• mysql数据导入hdfs

  bin/sqoop import \
  --connect jdbc:mysql://192.168.0.121:3306/gmall \
  --username root \
  --password root \
  --table user_info \
  --columns id,login_name \
  --target-dir /test \
  --delete-target-dir \
  --fields-terminated-by '\t' \
  --num-mappers 2 \
  --split-by id

• 第二种写法,可以直接写sql查询并导入

  bin/sqoop import \
  --connect jdbc:mysql://192.168.0.121:3306/gmall \
  --username root \
  --password root \
  --query 'select * from user_info where xxx and $CONDITIONS'
  --delete-target-dir \
  --fields-terminated-by '\t' \
  --num-mappers 2 \
  --split-by id

问题

sqoop lib包下添加mysql连接jar,common jar(必须是2.x版本)
mysql-connector-java-8.0.30.jar
commons-lang-2.6.jar