flume在k8s下程序发布

企业容器化部署成为趋势,基于k8s的容器编排成为中大企业部署服务的青睐。本文简单介绍k8s下flume程序的部署,flume一般用户大数据中数据的迁移以及数据接入。

基础镜像构建

基础镜像构建主要包括:
1.apache-flume包
2.mongoclients包:用于flume连mongo数据库
3.lib包:程序依赖
4.run.sh:程序启动脚本文件

基础镜像Dockerfile:

FROM openjdk:11.0.12-jre
# 准备环境
ENV LANGUAGE=zh_CN.UTF-8
ENV TZ=Asia/Shanghai
RUN mkdir /apps/
WORKDIR /apps/
# 添加文件
ADD apache-flume-1.9.0-bin /apps/apache-flume-1.9.0-bin
# 将 mongo-java-driver-2.6.5.jar、mongo-java-driver-3.12.7.jar
ADD mongoclients  /apps/mongoclients
# 将应用程序依赖拷贝到lib下
ADD lib  /lib/
# 将启动命令拷贝到apps下面
ADD run.sh /apps/

# 启动服务
ENV AGENT_NAME=a1
ENTRYPOINT ["/bin/sh","-c","bash run.sh"]

run.sh启动脚本:
启动脚本中会传递MONGO变量,用于指代我们mongo版本

#!/bin/bash
if [ $MONGO -eq 3 ]; then
    ln -sf /apps/mongoclients/mongo-java-driver-3.12.7.jar /lib/
elif [ $MONGO -eq 2 ]; then
    ln -sf /apps/mongoclients/mongo-java-driver-3.12.7.jar /lib/
fi

./apache-flume-1.9.0-bin/bin/flume-ng agent \
  -Xms2000m -Xmx4000m \
  --conf ./apache-flume-1.9.0-bin/conf \
  --conf-file ./apache-flume-1.9.0-bin/conf/flume.conf \
  -n ${AGENT_NAME:-a1}

工程镜像

我们的业务代码是依托于基础镜像框架,所以我们需要将我们的工程代码打包成一个依赖拷贝到flume应用程序框架中去。
我们将Dockerfile存放在我们的业务工程代码块中,然后通过mvn编译之后便可实现依赖打包镜像

Dockerfile:
xxx-1.0-SNAPSHOT.jar指代我们的业务代码。

FROM harbor.dc.prod:8000/base/flume:1.0
ADD target/xxx-1.0-SNAPSHOT.jar /lib/

构建镜像:

docker build -t harbor.dc.prod:8000/flume-access:1.0.0 .
docker push harbor.dc.prod:8000/flume-access:1.0.0

docker容器下服务启动

配置文件:

a1.channels = c1
a1.sources = s1
a1.sinks = k1

a1.sources.s1.type = 业务工程类型
a1.sources.s1.access.master.url = http://data-master:30001/source
a1.sources.s1.access.heartbeat.interval = 10
a1.sources.s1.access.supplier.sourceType = 业务渠道

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 80000000

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = kafka主题
a1.sinks.k1.kafka.bootstrap.servers = kafka1:9092,kafka2:9092,kafka3:9092,kafka4:9092,kafka5:9092
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.message.max.bytes = 102400000
a1.sinks.k1.kafka.consumer.max.partition.fetch.bytes = 102400000
a1.sinks.k1.kafka.producer.max.request.size = 102400000
a1.sinks.k1.flumeBatchSize = 20
a1.sinks.k1.channel = c1

a1.sources.s1.channels = c1

启动脚本:docker-run.sh

#!/bin/sh
version=$2
echo $version
NAME=flume-test
IMAGE="harbor.dc.prod:8000/flume-access:$version"
MONGO=3

SELF=`readlink -f $0`
D=`dirname $SELF`

if [ ! -d $D/logs ]; then
    mkdir -p $D/logs
fi

docker run -d --restart=always \
  -v $D/$NAME.conf:/apps/apache-flume-1.9.0-bin/conf/flume.conf \
  -v $D/logs:/apps/logs \
  -e MONGO=$MONGO \
  --name "flume-$NAME" "$IMAGE"

然后我们执行: