docker搭建kafka集群+ZK

docker搭建kafka集群+ZK

在写kafka之前，docker搭建一套kafka集群环境，用的腾讯云，这里真的不得不吐槽一下：腾讯云的体验比阿里云强，阿里云要不你反思反思？

1、创建子网，管理方便一点

docker network create --subnet 172.19.0.0/16 --gateway 172.19.0.1 zookeeper_kafka

2、创建docker-compose.yml文件，cat vi都可以

version: '3.3'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - 2181:2181
    container_name: zookeeper
    networks:
      default:
        ipv4_address: 172.19.0.11
  kafka0:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka0
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:9092
      KAFKA_LISTENERS: PLAINTEXT://kafka0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 0
    volumes:
      - /root/data/kafka0/data:/data
      - /root/data/kafka0/log:/datalog
    networks:
      default:
        ipv4_address: 172.19.0.12
  kafka1:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka1
    ports:
      - 9093:9093
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9093
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
    volumes:
      - /root/data/kafka1/data:/data
      - /root/data/kafka1/log:/datalog
    networks:
      default:
        ipv4_address: 172.19.0.13
  kafka2:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka2
    ports:
      - 9094:9094
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9094
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 2
    volumes:
      - /root/data/kafka2/data:/data
      - /root/data/kafka2/log:/datalog
    networks:
      default:
        ipv4_address: 172.19.0.14
  kafka3:
    image: wurstmeister/kafka
    depends_on:
      - zookeeper
    container_name: kafka3
    ports:
      - 9095:9095
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9095
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9095
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 3
    volumes:
      - /root/data/kafka3/data:/data
      - /root/data/kafka3/log:/datalog
    networks:
      default:
        ipv4_address: 172.19.0.15
  kafka-manager:
    image: sheepkiller/kafka-manager:latest
    restart: unless-stopped
    container_name: kafka-manager
    hostname: kafka-manager
    ports:
      - "9000:9000"
    links:            # 连接本compose文件创建的container
      - kafka1
      - kafka2
      - kafka3
    external_links:   # 连接本compose文件以外的container
      - zookeeper
    environment:
      ZK_HOSTS: zoo1:2181                 ## 修改:宿主机IP
      TZ: CST-8
networks:
  default:
    external:
      name: zookeeper_kafka

在没有端口占用的情况下可以直接复制使用

3、之后docker-compose启动，别再问这里-d啥意思了，不懂百度

docker-compose -f docker-compose.yml up -d

 截图一下成功的模样

 4、测试kafka生产者消费者

进入kafka0，

docker exec -it kafka0 /bin/bash

进入脚本文件目录

cd /opt/kafka_2.13-2.7.1/bin/

创建topic为chat的主题，

kafka-topics.sh --create --topic chat --partitions 5 --zookeeper 8.210.138.111:2181 --replication-factor 3

运行生产者脚本

kafka-console-producer.sh --broker-list kafka0:9092 --topic chat

再开一个客户端，进入kafka1，2，3都行，进入bin脚本文件目录

运行消费者脚本

kafka-console-consumer.sh --bootstrap-server kafka2:9094 --topic chat --from-beginning

在生产者窗口直接发送消息

 在消费者窗口能看到消息

 基本搭建就结束了

之后差一个kafka manager客户端

这玩意要用kubernates去安装，你要先安装kubernates，这里我们先埋个坑，因为我想用kibana去做一个类似kafka manager的东西，我们先用这样的集群已经没有问题了