k8s上zookeeper和kafka部署

k8s上zookeeper和kafka部署 helm安装kafka 1、helm安装

不做介绍

2、helm下载kafka和zk

[root@cce-master1 soft]# helm search hub kafka
URL                                               	CHART VERSION	APP VERSION    	DEscriptION
https://hub.helm.sh/charts/riftbit/kafka          	14.2.0       	2.8.1          	Apache Kafka is a distributed streaming platform.
https://hub.helm.sh/charts/bitnami/kafka          	14.9.0       	2.8.1          	Apache Kafka is a distributed streaming platform.
https://hub.helm.sh/charts/bitnami-aks/kafka      	14.9.0       	2.8.1          	Apache Kafka is a distributed streaming platform.


helm pull bitnami/kafka
helm pull bitnami/zookeeper

3、安装zookeeper

[root@cce-master1 soft]# helm install -n zookeeper zookeeper bitnami/zookeeper --set persistence.storageClass=copaddon-zk-kafka,persistence.size=1Gi,replicaCount=3

helm install -f zookeeper/values.yaml --name zookeeper-cluster bitnami/zookeeper --namespace zookeeper
[root@cce-master1 soft]#  helm install -n zookeeper zookeeper bitnami/zookeeper --set persistence.storageClass=copaddon-zk-kafka,persistence.size=1Gi,replicaCount=3
NAME: zookeeper
LAST DEPLOYED: Wed Jan 12 16:50:23 2022
NAMESPACE: zookeeper
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: zookeeper
CHART VERSION: 7.6.1
APP VERSION: 3.7.0

** Please be patient while the chart is being deployed **

ZooKeeper can be accessed via port 2181 on the following DNS name from within your cluster:

    zookeeper.zookeeper.svc.cluster.local

To connect to your ZooKeeper server run the following commands:

    export POD_NAME=$(kubectl get pods --namespace zookeeper -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}")
    kubectl exec -it $POD_NAME -- zkCli.sh

To connect to your ZooKeeper server from outside the cluster execute the following commands:

    kubectl port-forward --namespace zookeeper svc/zookeeper 2181:2181 &
    zkCli.sh 127.0.0.1:2181
    
    
或者是修该values.yaml文件，修改相应参数
persistence.storageClass=copaddon-zk-kafka
persistence.size=1Gi,replicaCount=3
global.imagePullSecrets= ["default-secret"]
image仓库
helm install -f zookeeper/values.yaml --name zookeeper-cluster bitnami/zookeeper --namespace zookeeper

[root@cce-master3 ~]# kubectl get pod -n zookeeper
NAME          READY   STATUS    RESTARTS   AGE
zookeeper-0   1/1     Running   0          51s
zookeeper-1   1/1     Running   0          51s
zookeeper-2   1/1     Running   0          49s

卸载
helm uninstall -n zookeeper zookeeper

4、安装kafka

zookeeper.zookeeper.svc.cluster.local

helm install -n kafka kafka bitnami/kafka --set zookeeper.enabled=false --set replicaCount=3 --set externalZookeeper.servers=zookeeper.zookeeper.svc.cluster.local --set persistence.storageClass=copaddon-zk-kafka

helm install -f kafka/values.yaml --name kafka-cluster bitnami/kafka --namespace kafka
[root@cce-master1 soft]# helm install -n kafka kafka bitnami/kafka --set zookeeper.enabled=false --set replicaCount=3 --set externalZookeeper.servers=zookeeper.zookeeper.svc.cluster.local --set persistence.storageClass=copaddon-zk-kafka
NAME: kafka
LAST DEPLOYED: Wed Jan 12 17:33:16 2022
NAMESPACE: kafka
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 14.9.0
APP VERSION: 2.8.1

** Please be patient while the chart is being deployed **

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

    kafka.kafka.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    kafka-0.kafka-headless.kafka.svc.cluster.local:9092
    kafka-1.kafka-headless.kafka.svc.cluster.local:9092
    kafka-2.kafka-headless.kafka.svc.cluster.local:9092

To create a pod that you can use as a Kafka client run the following commands:

    kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.1-debian-10-r73 --namespace kafka --command -- sleep infinity
    kubectl exec --tty -i kafka-client --namespace kafka -- bash

    PRODUCER:
        kafka-console-producer.sh 

            --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 
            --topic test

    CONSUMER:
        kafka-console-consumer.sh 

            --bootstrap-server kafka.kafka.svc.cluster.local:9092 
            --topic test 
            --from-beginning
            
            
或者修改values.yaml文件，修改成桉树
replicaCount=3
externalZookeeper.servers=zookeeper.zookeeper.svc.cluster.local
persistence.storageClass=copaddon-zk-kafka
global.imagePullSecrets= ["default-secret"]
image修改
helm install -f kafka/values.yaml --name kafka-cluster bitnami/kafka --namespace kafka

5、zookeeper测试

kubectl get pods --namespace zookeeper -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}"

export POD_NAME=$(kubectl get pods --namespace zookeeper -l "app.kubernetes.io/name=zookeeper,app.kubernetes.io/instance=zookeeper,app.kubernetes.io/component=zookeeper" -o jsonpath="{.items[0].metadata.name}")

kubectl exec -it $POD_NAME -n zookeeper -- zkCli.sh #可以查看zk中相关的目录

6、kafka测试

kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:2.8.1-debian-10-r73 --namespace kafka --command -- sleep infinity
kubectl get pod -n kafka
kubectl exec --tty -i kafka-client --namespace kafka -- bash

PRODUCER:
kafka-console-producer.sh --broker-list kafka-0.kafka-headless.kafka.svc.cluster.local:9092,kafka-1.kafka-headless.kafka.svc.cluster.local:9092,kafka-2.kafka-headless.kafka.svc.cluster.local:9092 --topic test

ConSUMER
kafka-console-consumer.sh --bootstrap-server kafka.kafka.svc.cluster.local:9092 --topic test --from-beginning

7、kafka-manager安装

[root@cce-master1 soft]# helm repo add stable http://mirror.azure.cn/kubernetes/charts
[root@cce-master1 soft]# helm search repo kafka-manager
NAME                	CHART VERSION	APP VERSION	DEscriptION
stable/kafka-manager	2.3.5        	1.3.3.22   	DEPRECATED - A tool for managing Apache Kafka.
[root@cce-master1 soft]# helm fetch stable/kafka-manager
[root@cce-master1 soft]# cat kafka-manager-values.yaml
image:
  repository: cce.zhanghsn/com/zhanghsn/kafka-manager
  tag: 1.3.3.22
zkHosts: zookeeper:2181
basicAuth:
  enabled: true
  username: admin
  password: admin
ingress:
  enabled: true
  hosts:
   - km.hongda.com

helm install --name kafka-manager --namespace kafka -f kafka-manager-values.yaml stable/kafka-manager

[root@cce-master1 soft]# kubectl get pod,svc,ing -nkafka
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME                                 READY   STATUS    RESTARTS   AGE
pod/kafka-0                          1/1     Running   0          104m
pod/kafka-1                          1/1     Running   0          104m
pod/kafka-2                          1/1     Running   0          104m
pod/kafka-client                     1/1     Running   0          94m
pod/kafka-manager-7869d5f76b-jtp8f   1/1     Running   0          7m42s

NAME                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/kafka                    ClusterIP   10.247.42.103            9092/TCP            17h
service/kafka-headless           ClusterIP   None                     9092/TCP,9093/TCP   17h
service/kafka-manager            ClusterIP   10.247.149.131           9000/TCP            7m42s
service/kafka-manager-nodeport   NodePort    10.247.43.39             9000:32065/TCP      21m

NAME                               CLASS    HOSTS           ADDRESS                                  PORTS   AGE
ingress.extensions/kafka-manager      km.hongda.com   10.11.95.235,10.11.95.236,10.11.95.237   8	0      7m42s
[root@cce-master1 soft]#

添加kafka集群失败，问题还有有待解决