借助 Docker 在 Arm 设备上运行 Hbase

之前在 x86 平台上，使用 harisekhon/hbase 一键跑 hbase 用于测试开发

但是发现在 arm 上，这样跑会出现 TTransportException(type=4, message='TSocket read 0 bytes') 错误

而且 harisekhon/hbase 也没有适配 arm，只有 x86

所以就自己动手制作一个可以跑在 arm 上的 hbase 吧

---

先准备四个文件：

• Makefile 这玩意的作用，参考：使用 makefile 管理 docker-compose
• entrypoint.sh 这个是帮我们一次性启动 hbase 以及 thrift 的
• Dockerfile 打包镜像
• docker-compose.yaml 运行 container from image

这里只包含最小能跑的 hbase，包含一个 thrift，其他的 zookeeper 不包含

Makefile

NAME = ponponon/hbase-arm-focal
VERSION = 0.0.1

.PHONY: build up stop logs

build:  docker-build
up: docker-compose-up
stop: docker-compose-stop
logs: docker-compose-logs

docker-build:
    docker build -t "${NAME}" .

docker-compose-up:
    docker-compose up -d

docker-compose-stop:
    docker-compose stop

docker-compose-logs:
    docker-compose logs --tail=100 -f

entrypoint.sh

#!/usr/bin/env bash
#  vim:ts=4:sts=4:sw=4:et
#
#  Author: Hari Sekhon
#  Date: 2016-04-24 21:29:46 +0100 (Sun, 24 Apr 2016)
#
#  https://github.com/harisekhon/Dockerfiles
#
#  License: see accompanying Hari Sekhon LICENSE file
#
#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback
#
#  https://www.linkedin.com/in/harisekhon
#

set -x
set -euo pipefail
[ -n "${DEBUG:-}" ] && set -x


export HBASE_HOME="/root/hbase-2.4.17"
export JAVA_HOME="${JAVA_HOME:-/usr}"

echo "================================================================================"
echo "                              HBase Docker Container"
echo "================================================================================"
echo
# shell breaks and doesn't run zookeeper without this
mkdir -pv "$HBASE_HOME/logs"

start_zookeeper(){
    # tries to run zookeepers.sh distributed via SSH, run zookeeper manually instead now
    #RUN sed -i 's/# export HBASE_MANAGES_ZK=true/export HBASE_MANAGES_ZK=true/' "$HBASE_HOME/conf/hbase-env.sh"
    echo
    echo "Starting Zookeeper..."
    "$HBASE_HOME/bin/hbase" zookeeper &>"$HBASE_HOME/logs/zookeeper.log" &
    echo
}

start_master(){
    echo "Starting HBase Master..."
    "$HBASE_HOME/bin/hbase-daemon.sh" start master
    echo
}

start_regionserver(){
    echo "Starting HBase RegionServer..."
    # HBase versions < 1.0 fail to start RegionServer without SSH being installed
    if [ "$(echo /hbase-* | sed 's,/hbase-,,' | cut -c 1)" = 0 ]; then
        "$HBASE_HOME/bin/local-regionservers.sh" start 1
    else
        "$HBASE_HOME/bin/hbase-daemon.sh" start regionserver
    fi
    echo
}

start_stargate(){
    # kill any pre-existing rest instances before starting new ones
    pgrep -f proc_rest && pkill -9 -f proc_rest
    echo "Starting HBase Stargate Rest API server..."
    "$HBASE_HOME/bin/hbase-daemon.sh" start rest
    echo
}

start_thrift(){
    # kill any pre-existing thrift instances before starting new ones
    pgrep -f proc_thrift && pkill -9 -f proc_thrift
    echo "Starting HBase Thrift API server..."
    "$HBASE_HOME/bin/hbase-daemon.sh" start thrift
    #"$HBASE_HOME/bin/hbase-daemon.sh" start thrift2
    echo
}

start_hbase_shell(){
    echo "
Example Usage:

create 'table1', 'columnfamily1'

put 'table1', 'row1', 'columnfamily1:column1', 'value1'

get 'table1', 'row1'


Now starting HBase Shell...
"
    "$HBASE_HOME/bin/hbase" shell
}

trap_func(){
    echo -e "\n\nShutting down HBase:"
    "$HBASE_HOME/bin/hbase-daemon.sh" stop rest || :
    "$HBASE_HOME/bin/hbase-daemon.sh" stop thrift || :
    "$HBASE_HOME/bin/local-regionservers.sh" stop 1 || :
    # let's not confuse users with superficial errors in the Apache HBase scripts
    "$HBASE_HOME/bin/stop-hbase.sh" |
        grep -v -e "ssh: command not found" \
                -e "kill: you need to specify whom to kill" \
                -e "kill: can't kill pid .*: No such process"
    sleep 2
    pgrep -fla org.apache.hadoop.hbase |
        grep -vi org.apache.hadoop.hbase.zookeeper |
            awk '{print $1}' |
                xargs kill 2>/dev/null || :
    sleep 3
    pkill -f org.apache.hadoop.hbase.zookeeper 2>/dev/null || :
    sleep 2
}
trap trap_func INT QUIT TRAP ABRT TERM EXIT

if [ -n "$*" ]; then
    if [ "$1" = master ] || [ "$1" = m ]; then
        start_master
        tail -f /dev/null "$HBASE_HOME/logs/"* &
    elif [ "$1" = regionserver ] || [ "$1" = rs ]; then
        start_regionserver
        tail -f /dev/null "$HBASE_HOME/logs/"* &
    elif [ "$1" = rest ] || [ "$1" = stargate ]; then
        start_stargate
        tail -f /dev/null "$HBASE_HOME/logs/"* &
    elif [ "$1" = thrift ]; then
        start_thrift
        tail -f /dev/null "$HBASE_HOME/logs/"* &
    elif [ "$1" = shell ]; then
        "$HBASE_HOME/bin/hbase" shell
    elif [ "$1" = bash ]; then
        bash
    else
        echo "usage:  must specify one of: master, regionserver, thrift, rest, shell, bash"
    fi
else
    sed -i 's/zookeeper:2181/localhost:2181/' "$HBASE_HOME/conf/hbase-site.xml"
    # start_zookeeper
    start_master
    # start_regionserver
    # start_stargate
    start_thrift
    if [ -t 0 ]; then
        start_hbase_shell
    else
        echo "
    Running non-interactively, will not open HBase shell

    For HBase shell start this image with 'docker run -t -i' switches
    "
        tail -f /dev/null "$HBASE_HOME/logs/"* &
        # this shuts down from Control-C but exits prematurely, even when +euo pipefail and doesn't shut down HBase
        # so I rely on the sig trap handler above
    fi
fi
wait || :

Dockerfile

FROM ubuntu:focal
RUN . /etc/os-release && cat > /etc/apt/sources.list <<EOF
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-updates main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-backports main restricted universe multiverse
deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ focal-security main restricted universe multiverse
EOF

RUN apt-get update
RUN apt-get install -y vim
RUN apt-get install -y openjdk-8-jdk

RUN apt-get install -y wget
WORKDIR /root
RUN wget -P /root https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/2.4.17/hbase-2.4.17-bin.tar.gz
RUN tar xvf /root/hbase-2.4.17-bin.tar.gz


RUN echo "export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64" | tee -a /etc/profile >/dev/null

COPY entrypoint.sh /root/

docker-compose.yaml

version: "3"
services:
  hbase-master:
    container_name: hbase-master
    image: ponponon/hbase-arm-focal
    # restart: always
    ports:
      - "2181:2181"
      - "8080:8080"
      - "8085:8085"
      - "9090:9090"
      - "9095:9095"
      - "16000:16000"
      - "16010:16010"
      - "16201:16201"
      - "16020:16020"
      - "16030:16030"
      - "16301:16301"
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64
    command: 'bash entrypoint.sh'
  # hbase-thrift:
  #   container_name: hbase-thrift
  #   image: ponponon/hbase-x86-focal
  #   # restart: always
  #   ports:
  #     - "9090:9090"
  #   environment:
  #     - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
  #   command: 'bash /root/hbase-2.4.17/bin/hbase-daemon.sh start thrift --bind hbase-master'

准备好了四个文件之后，先运行

make build

docker-compose up

然后可以用下面的 python 脚本检查 hbase 是否可以正常运行

import happybase

# 定义HBase连接信息
host = '127.0.0.1'  # HBase主机名或IP地址
port = 9090  # HBase端口，默认为9090


# 建立HBase连接
connection = happybase.Connection(host=host, port=port)


# 获取所有表名
table_names = connection.tables()

# 删除所有表
print('删除所有表')
for table_name in table_names:
    connection.delete_table(table_name, disable=True)


table_name = 'my_table'  # 表名
column_families = {
    'cf1': dict(),  # 第一个列族
    'cf2': dict()   # 第二个列族
}

# 建立HBase连接
connection = happybase.Connection(host=host, port=port)

# 创建表
print('创建表')
connection.create_table(table_name, column_families)

# 获取表对象
table = connection.table(table_name)

# 插入数据
print('插入数据')
row_key = 'row1'
data = {
    'cf1:column1': 'value1',
    'cf1:column2': 'value2',
    'cf2:column3': 'value3'
}
table.put(row_key, data)

# 获取数据
row = table.row(row_key)
print(row)

# 扫描数据
print('读取数据')
scan_result = table.scan(row_start=row_key, row_stop=row_key)
for key, value in scan_result:
    print(key, value)

# 删除数据
table.delete(row_key)

# 关闭连接
connection.close()

输出下面的内容就 ok 了

╰─➤  python -u "/Users/ponponon/Desktop/code/me/ideaboom/5.py"
删除所有表
创建表
插入数据
{b'cf1:column1': b'value1', b'cf1:column2': b'value2', b'cf2:column3': b'value3'}
读取数据
b'row1' {b'cf1:column1': b'value1', b'cf1:column2': b'value2', b'cf2:column3': b'value3'}

---

我已经把上面的打包成 docker image 上传到 dockerhub 了

https://hub.docker.com/r/ponponon/hbase/tags

如果你要使用 hbase arm 版本，直接 docker pull ponponon/hbase:2.4.17 就行