【写在前面】飞腾开发者平台是基于飞腾自身强大的技术基础和开放能力，聚合行业内优秀资源而打造的。该平台覆盖了操作系统、算法、数据库、安全、平台工具、虚拟化、存储、网络、固件等多个前沿技术领域，包含了应用使能套件、软件仓库、软件支持、软件适配认证四大板块，旨在共享尖端技术，为开发者提供一个涵盖多领域的开发平台和工具套件。点击这里开始你的技术升级之旅吧
本文分享至飞腾开发者平台《飞腾平台Sqoop1.99.7移植与安装手册》
1 介绍
Sqoop（SQL–to–Hadoop）是一个用来将关系型数据库和Hadoop中的数据进行相互转移的工具，可以将一个关系型数据库(例如Mysql、Oracle)中的数据导入到Hadoop(例如HDFS、Hive、Hbase)中，也可以将Hadoop(例如HDFS Hive、Hbase)中的数据导入到关系型数据库(例如Mysql、Oracle)中。
本文主要介绍移植适配后的Sqoop1.99.7在飞腾平台的安装与部署过程。
2 环境要求
2.1 硬件要求
硬件要求如下表所示。
| 项目 | 说明 |
|---|---|
| CPU | FT-2000+/64服务器 |
| 网络 | 无要求 |
| 存储 | 无要求 |
| 内存 | 无要求 |
2.2 操作系统要求
操作系统要求如下表所示。
| 项目 | 说明 |
|---|---|
| CentOS | 8 |
| Kernel | 4.18.0-193.el8.aarch64 |
2.3 软件要求
软件要求如下表所示。
| 项目 | 说明 |
|---|---|
| Java | 1.8.0_281 |
| MySQL | 5.7 |
| Hadoop | 3.3.0 |
3 安装与部署
3.1 程序部署
下载apache-sqoop
wget http://archive.apache.org/dist/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz
mv sqoop-1.99.7-bin-hadoop200.tar.gz /opt
cd /opt/
tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
mv sqoop-1.99.7-bin-hadoop200.tar.gz sqoop-1.99.73.2 程序配置
1）配置环境变量
编辑 /etc/profile 文件，添加以下内容:
export SQOOP_HOME=/opt/sqoop-1.99.7
export PATH=$PATH:$SQOOP _HOME/bin2）配置日志信息
# vim /opt/sqoop-1.99.7/conf/sqoop.properties
#日志配置
BASEDIR=/opt/sqoop-1.99.7
LOGDIR=/opt/sqoop-1.99.7/logs
org.apache.sqoop.log4j.appender.file.File=/opt/sqoop-1.99.7/logs/sqoop.log
org.apache.sqoop.repository.sysprop.derby.stream.error.file=/opt/sqoop-1.99.7/logs/derbyrepo.log
# hadoop配置
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/opt/hadoop-3.3.0/etc/hadoop
#授权配置
org.apache.sqoop.security.authentication.type=SIMPLE
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler
org.apache.sqoop.security.authentication.anonymous=true3）进入$ SQOOP_HOME目录，并新建 conf/hive-site.xml 文件添加以下配置
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>hadoop</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>hadoop</value>
</property>4）上传Mysql驱动包到hive的lib目录
cp mysql-connector-java.jar /opt/sqoop-1.99.7/server/lib/3.3 启动服务
1）验证服务
[root@master sqoop-1.99.7]# sqoop2-tool verify
Setting conf dir: /usr/local/apps/sqoop-1.99.7/bin/../conf
Sqoop home directory: /usr/local/apps/sqoop-1.99.7
Sqoop tool executor:
Version: 1.99.7
Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0 [main] INFO org.apache.sqoop.core.SqoopServer – Initializing Sqoop server.
41 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider – Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/apps/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/apps/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:37)
at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
at org.apache.sqoop.tools.tool.VerifyTool.runTool(VerifyTool.java:36)
at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)以上报错，经过分析后发现是guava版本问题，qoop内部依赖guava版本与hadoop依赖的guava版本存在冲突与兼容性问题。
解决方案：
[root@master sqoop-1.99.7-bin-hadoop200]# find /opt/sqoop-1.99.7/ -name guava*.jar
./tools/lib/guava-11.0.2.jar
./shell/lib/guava-11.0.2.jar
./server/lib/guava-11.0.2.jar
[root@master sqoop-1.99.7-bin-hadoop200]#
[root@master sqoop-1.99.7-bin-hadoop200]
[root@master sqoop-1.99.7-bin-hadoop200]# ls /opt/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar
/opt/hadoop-3.3.0/share/hadoop/common/lib/guava-27.0-jre.jar
用hadoop的版本文件替换 /opt/sqoop-1.99.7/ 目录下的所有guava文件再次验证：
[hadoop@master sqoop-1.99.7]$ sqoop2-tool verify
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop tool executor:
Version: 1.99.7
Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
13 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.2）升级检测
# bin/sqoop2-tool upgrader结果显示：
[hadoop@master sqoop-1.99.7]# bin/sqoop2-tool upgrade
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.sqoop.utils.ClassUtils.loadClassWithClassLoader(ClassUtils.java:106)
at org.apache.sqoop.utils.ClassUtils.loadClass(ClassUtils.java:70)
at org.apache.sqoop.repository.JdbcRepositoryProvider.initializeRepositoryHandler(JdbcRepositoryProvider.java:122)
at org.apache.sqoop.repository.JdbcRepositoryProvider.initialize(JdbcRepositoryProvider.java:65)
at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:124)
at org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:39)
at org.apache.sqoop.tools.ConfiguredTool.runTool(ConfiguredTool.java:35)
at org.apache.sqoop.tools.ToolRunner.main(ToolRunner.java:72)
Caused by: java.lang.SecurityException: sealing violation: package org.apache.derby.impl.jdbc.authentication is sealed
at java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:400)
at java.net.URLClassLoader.definePackageInternal(URLClassLoader.java:420)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:452)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)以上报错，经过分析后发现是guava版本问题，qoop内部依赖derby版本与hive版本依赖的derby版本存在一致性问题。
解决方案：
[root@master sqoop-1.99.7]# find . -name derby*.jar
./server/lib/derby-10.8.2.2.jar
[root@master sqoop-1.99.7]# find /opt/hive-3.1.2/ -name derby*.jar
LICENSE RELEASE_NOTES.txt bin/ conf/ hcatalog/ lib/
NOTICE auxlib/ binary-package-licenses/ examples/ jdbc/ scripts/
[root@master sqoop-1.99.7]# find /opt/hive-3.1.2/lib/ -name derby*.jar
/opt/hive-3.1.2/lib/derby-10.14.1.0.jar
[root@master sqoop-1.99.7]# mv server/lib/derby-10.8.2.2.jar server/lib/derby-10.8.2.2.jar.old
[root@master sqoop-1.99.7]# cp /opt/hive-3.1.2/lib/derby-10.14.1.0.jar server/lib/再次验证：
[hadoop@master sqoop-1.99.7]$ bin/sqoop2-tool upgrade
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop tool executor:
Version: 1.99.7
Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
0 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.3）启动服务
[hadoop@master sqoop-1.99.7]$ /opt/sqoop-1.99.7/bin/sqoop.sh server start
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Starting the Sqoop2 server...
0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
9 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop-3.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Sqoop2 server started.检测端口服务：
[hadoop@master opt]$ sudo netstat -anpt|grep 1137694
tcp6 0 0 :::12000 :::* LISTEN
1137694/java端口与服务均正常，表示sqoop启动正常。
4 功能测试
4.1 启动客户端测试
1）运行sqoop2客户端命令
[hadoop@master sqoop-1.99.7]$ sqoop2-shell
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop Shell: Type 'help' or '\h' for help.
[ERROR] Failed to construct terminal; falling back to unsupported
java.lang.NumberFormatException: For input string: "0x100"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at jline.internal.InfoCmp.parseInfoCmp(InfoCmp.java:59)
at jline.UnixTerminal.parseInfoCmp(UnixTerminal.java:233)
at jline.UnixTerminal.<init>(UnixTerminal.java:64)
at jline.UnixTerminal.<init>(UnixTerminal.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at jline.TerminalFactory.getFlavor(TerminalFactory.java:209)
at jline.TerminalFactory.create(TerminalFactory.java:100)
at jline.TerminalFactory.get(TerminalFactory.java:184)启动客户端报错，解决方案如下：
[hadoop@master sqoop-1.99.7]$ export TERM=xterm-color再次运行sqoop2客户端命令
[hadoop@master sqoop-1.99.7]$ $SQOOP_HOME/bin/sqoop.sh client
Setting conf dir: /opt/sqoop-1.99.7/bin/../conf
Sqoop home directory: /opt/sqoop-1.99.7
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>2）连接sqoop2-server
sqoop:000> set server --host localhost --port 12000 --webapp sqoop
Server is set successfully3）查看当前sqoop2所有连接
sqoop:000> show connector
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+------------------------+---------+------------------------------------------------------------+----------------------+
| Name | Version | Class | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO |
| kite-connector | 1.99.7 | org.apache.sqoop.connector.kite.KiteConnector | FROM/TO |
| oracle-jdbc-connector | 1.99.7 | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO |
| ftp-connector | 1.99.7 | org.apache.sqoop.connector.ftp.FtpConnector | TO |
| hdfs-connector | 1.99.7 | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO |
| kafka-connector | 1.99.7 | org.apache.sqoop.connector.kafka.KafkaConnector | TO |
| sftp-connector | 1.99.7 | org.apache.sqoop.connector.sftp.SftpConnector | TO |
+------------------------+---------+------------------------------------------------------------+----------------------+结果表明：Sqoop2程序在飞腾平台下运行结果正确，符合预期，功能正常。
推荐阅读
欢迎广大开发者来飞腾开发者平台获取更多前沿技术文档及资料
如开发者在使用飞腾产品有任何问题可通过在线工单联系我们
版权所有。飞腾信息技术有限公司 2023。保留所有权利。
未经本公司同意，任何单位、公司或个人不得擅自复制，翻译，摘抄本文档内容的部分或全部，不得以任何方式或途径进行传播和宣传。
商标声明
Phytium和其他飞腾商标均为飞腾信息技术有限公司的商标。
本文档提及的其他所有商标或注册商标，由各自的所有人拥有。
注意
本文档的内容视为飞腾的保密信息，您应当严格遵守保密任务；未经飞腾事先书面同意，您不得向任何第三方披露本文档内容或提供给任何第三方使用。
由于产品版本升级或其他原因，本文档内容会不定期进行更新。除非另有约定，本文档仅作为使用指导，飞腾在现有技术的基础上尽最大努力提供相应的介绍及操作指引，但飞腾在此明确声明对本文档内容的准确性、完整性、适用性、可靠性的等不作任何明示或暗示的保证。
本文档中所有内容，包括但不限于图片、架构设计、页面布局、文字描述，均由飞腾和/或其关联公司依法拥有其知识产权，包括但不限于商标权、专利权、著作权等。非经飞腾和/或其关联公司书面同意，任何人不得擅自使用、修改，复制上述内容。
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。