HBase2.1.10源码导入Idea本地调试

一、环境准备

环境准备
1. Jdk 11

2. maven 3.6.3

3. Idea 2020.1

4. Git 2.24.3

5. Mac OS

二.克隆源码到本地目录

git clone https://github.com/apache/hbase

三.使用Idea导入项目并配置jdk

3.1 切换需要编译的分支，此处我选择的是2.1.10

3.2 导入到idea后的jdk版本需要做调整，让每一个模块都加载jdk11

3.3 修改根目录下的pom.xml，修改jdk版本、修改maven插件版本、跳过license 检测

<properties>
    <compileSource>11</compileSource>
    <maven.compiler.version>3.8.1</maven.compiler.version>
</properties>

<plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>${maven.compiler.version}</version>
          <configuration>
            <source>${compileSource}</source>
            <target>${compileSource}</target>
            <showWarnings>true</showWarnings>
            <showDeprecation>false</showDeprecation>
            <useIncrementalCompilation>false</useIncrementalCompilation>
            <!-- <compilerArgument>-Xlint:-options</compilerArgument> -->
            <compilerArgs>
              <arg>--add-exports=java.base/jdk.internal.access=ALL-UNNAMED</arg>
              <arg>--add-exports=java.base/jdk.internal=ALL-UNNAMED</arg>
              <arg>--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED</arg>
              <arg>--add-exports=java.base/sun.security.pkcs=ALL-UNNAMED</arg>
              <arg>--add-exports=java.base/sun.nio.ch=ALL-UNNAMED</arg>
            </compilerArgs>
          </configuration>
        </plugin>

<!-- 跳过license检查 -->
<execution>
            <id>check-aggregate-license</id>
            <!-- must check after LICENSE is built at 'generate-resources' -->
            <phase>process-resources</phase>
            <goals>
              <goal>enforce</goal>
            </goals>
            <configuration>
              <rules>
                <evaluateBeanshell>
                  <condition>
                    File license = new File("${license.aggregate.path}");

                    // Beanshell does not support try-with-resources,
                    // so we must close this scanner manually
                    Scanner scanner = new Scanner(license);

                    while (scanner.hasNextLine()) {
                    if (scanner.nextLine().startsWith("ERROR:")) {
                    scanner.close();
                    return false;
                    }
                    }
                    scanner.close();
                    return true;
                  </condition>
                  <message>
                    License errors detected, for more detail find ERROR in
                    ${license.aggregate.path}
                  </message>
                </evaluateBeanshell>
              </rules>
              <!--<skip>${skip.license.check}</skip>-->
              <skip>true</skip>
            </configuration>
</execution>

修改后：

4.执行maven打包命令

mvn clean package -DskipTests=true

5.编译提示javax.annotation不存在

Error:(6,18) java: 程序包javax.annotation不存在

5.1 解决办法：修改对应报错类的pom文件，添加对应依赖（hbase-protocol-shaded下的pom.xml），hbase-protocol、根目录下的pom.xml也需要添加

<dependency>
      <groupId>javax.annotation</groupId>
      <artifactId>javax.annotation-api</artifactId>
      <version>1.3.1</version>
</dependency>

6 重新编译，再次报错

Error:(34,25) java: 程序包javax.xml.ws.http不存在

6.1 解决办法:在hbase-it项目的pom文件中添加依赖，需要添加到最外层的dependencies中

<dependency>
      <groupId>jakarta.xml.ws</groupId>
      <artifactId>jakarta.xml.ws-api</artifactId>
      <version>2.3.3</version>
</dependency>

6.2 再次执行编译命令，继续报错

Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (default) on project hbase-protocol-shaded: Error creating shaded jar: null

6.3 解决办法，修改各模块中引用maven-shade-plugin的版本为3.2.4，需要修改的模块有

/hbase/hbase-shaded/pom.xml
/hbase/hbase-protocol-shaded/pom.xml
/hbase/hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml
/hbase/hbase-shaded/hbase-shaded-mapreduce/pom.xml
/hbase/hbase-shaded/hbase-shaded-client/pom.xml

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
</plugin>

6.4 再次执行编译命令，终于编译成功

7. 为HBase-Server模块添加配置文件

7.1 hbase-site.xml、log4j.propertiies，从其它项目拷贝过来即可，修改对应配置

  <property>
    <name>hbase.rootdir</name>
 <value>file:///Users/admin/Documents/software/data/hbase/hbase_data</value>
    <description>hbase本地数据地址</description>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/Users/admin/Documents/software/data/hbase/zookeeper-data</value>
    <description>hbase 内置zk数据目录地址</description>
  </property>
  <property>
    <name>hbase.defaults.for.version.skip</name>
    <value>true</value>
    <description>跳过版本检查</description>
  </property>

8.启动HMaster

8.1 Run -> Edit Configurations -> 左上角+ -> Application，填入配置信息

VM options 具体配置：
-Dhbase.home.dir=/Users/admin/Documents/software/hbase/hbase
-Dhbase.id.str=root
-Dlog4j.configuration=file:///Users/admin/Documents/software/hbase/hbase-server/src/main/resources/log4j.properties
-Dhbase.log.dir=/Users/admin/Documents/software/data/hbase/hbase_log
-Dhbase.log.file=hbase-root-master.log
-Dhbase.root.logger=INFO,console,DRFA
--add-exports=java.base/jdk.internal.access=ALL-UNNAMED
--add-exports=java.base/jdk.internal=ALL-UNNAMED
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
--add-exports=java.base/sun.security.pkcs=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens
java.base/jdk.internal.misc=ALL-UNNAMED
-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible=true

8.2 启动HMaster

8.3 再次报错

Error:(56, 16) java: 程序包sun.misc不存在

8.3.1 解决办法，更换引用包，替换包名称，需要替换以下几个类中的包

org.apache.hadoop.hbase.util.Bytes
//import sun.misc.Unsafe;
import jdk.internal.misc.Unsafe;

org.apache.hadoop.hbase.util.UnsafeAccess
//import sun.misc.Unsafe;
import jdk.internal.misc.Unsafe;

org.apache.hadoop.hbase.io.hfile.bucket.UnsafeSharedMemoryBucketEntry
//import sun.misc.Unsafe;
import jdk.internal.misc.Unsafe;

9. 启动HMaster

启动成功。访问 localhost:16010

10. 配置HBase Shell模块，开启本地shell调试

10.1 Run -> Edit Configurations -> 左上角+ -> Application，填入配置信息

具体参数配置：
main class: org.jruby.Main
VM options: 
-Dhbase.ruby.sources=/Users/admin/Documents/software/hbase/hbase-shell/src/main/ruby
--add-exports=java.base/jdk.internal.access=ALL-UNNAMED
--add-exports=java.base/jdk.internal=ALL-UNNAMED
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
--add-exports=java.base/sun.security.pkcs=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens
java.base/jdk.internal.misc=ALL-UNNAMED

Program arguments: /Users/admin/Documents/software/hbase/bin/hirb.rb

11. 启动HBase shell，大功告成。可以在控制台进行命令操作