Kafka mirror

Kafka mirror 准备两个Kafka集群 （default | loganalysis）

配置文件

### loganalysis kafka-0 ###
root@kafka-0:/etc/kafka# ls
connect-console-sink.properties    connect-file-sink.properties    connect-standalone.properties  log4j.properties     server.properties           tools-log4j.properties
connect-console-source.properties  connect-file-source.properties  consumer.properties            producer.properties  source-consumer.properties  trogdor.conf
connect-distributed.properties     connect-log4j.properties        kafka.properties               secrets              target-producer.properties  zookeeper.properties

### 数据源 Kafka ###

root@kafka-0:/etc/kafka# cat source-consumer.properties
bootstrap.servers=kafka-headless:9092
zookeeper.connect=kafka-zookeeper-headless:2181
zookeeper.connection.timeout.ms=1000000
group.id=mirror-maker-group-test
message.max.bytes=7000000
fetch.message.max.bytes=7500000
max.partition.fetch.bytes=7500000
replica.fetch.max.bytes=7500000

### 目标 Kafka ###

root@kafka-0:/etc/kafka# cat target-producer.properties
bootstrap.servers=loganalysis-kafka-headless.default:9092
message.max.bytes=7000000
fetch.message.max.bytes=7500000
max.partition.fetch.bytes=7500000
replica.fetch.max.bytes=7500000
root@kafka-0:/etc/kafka# 

kafka-mirror

root@kafka-2:/# kafka-mirror-maker
Continuously copy data between two Kafka clusters.
Option                                   Description
------                                   -----------
--abort.on.send.failure 
--consumer.config   Embedded consumer config for consuming
                                           from the source cluster.
--consumer.rebalance.listener 
--help                                   Print this message.
--message.handler                producer.
--message.handler.args 
--new.consumer                           DEPRECATED Use new consumer in mirror
                                           maker (this is the default so this
                                           option will be removed in a future
                                           version).
--num.streams                                  (default: 1)
--offset.commit.interval.ms 
--producer.config   Embedded producer config.
--rebalance.listener.args 
--whitelist 
root@kafka-2:/#

root@kafka-0:/etc/kafka# kafka-mirror-maker --consumer.config source-consumer.properties --producer.config target-producer.properties  --whitelist test

kafka集群1

kafka集群2