使用docker-compose部署kafka服务

使用Docker Compose部署Kafka消息服务

• 关于Kafka的介绍，可以参考对于Kafka的基本认识

安装

• Docker Compose 安装

• 部署服务

  • Kafka
  • Zookeeper
  • Kafka Manager

# 1. 安装docker-compose，需要预先安装好Docker
sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
# 增加可执行权限
sudo chmod +x /usr/local/bin/docker-compose
# 2. 创建相关文件夹
mkdir -p /opt/zookeeper && mkdir -p /opt/kafka
# 3. 准备配置文件
echo '
version: '3.7'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
       - /opt/zookeeper/data:/data
    container_name: zookeeper
    mem_limit: 1024M
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181
    restart: always
  kafka_node1:
    image: wurstmeister/kafka
    container_name: kafka_node1
    mem_limit: 1024M
    depends_on:
      - zookeeper
    ports: 
      - 9092:9092
    volumes:
      - /opt/kafka/data:/kafka
    environment:
      KAFKA_CREATE_TOPICS: "test"
      KAFKA_BROKER_NO: 0
      KAFKA_LISTENERS: PLAINTEXT://kafka_node1:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://${kafka_service_public_ip}:${kafka_service_public_port}
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_HEAP_OPTS: "-Xmx512M -Xms16M"
    restart: always
  kafka_manager: 
    image: hlebalbau/kafka-manager:stable
    ports:
      - 9000:9000
    environment:
      ZK_HOSTS: "zookeeper:2181"  
    depends_on:
      - zookeeper
      - kafka_node1
    restart: always' > /etc/kafka/kafka.yml
# 3. 启动容器服务
docker-compose -f /etc/kafka/kafka.yml up -d

• 注意修改KAFKA_ADVERTISED_LISTENERS环境变量中的服务IP和端口，一般是容器所在宿主机的可访问IP与9092端口

使用Kafka-Mgr管理集群

• 访问服务所在主机的9000端口，按如下图所示配置集群：

• 若出现 Yikes! KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again. 错误，参考此issue解决，具体步骤如下：

  docker exec -it zookeeper bash
  ./bin/zkCli.sh
  [zk: localhost:2181(CONNECTED) 2] ls /kafka-manager[configs, deleteClusters, clusters]
  [zk: localhost:2181(CONNECTED) 3] create /kafka-manager/mutex ""Created /kafka-manager/mutex
  [zk: localhost:2181(CONNECTED) 5] create /kafka-manager/mutex/locks ""Created /kafka-manager/mutex/locks
  [zk: localhost:2181(CONNECTED) 6] create /kafka-manager/mutex/leases ""Created /kafka-manager/mutex/leases

参考

• Docker环境下部署Kafka
• docker-compose部署Kafka
• Kafka-Mgr的使用