Elasticsearch & Kibana & Filebeat开启SSL通信

Elasticsearch & Kibana & Filebeat开启SSL通信

EFK搭建并开启TLS通信

• 搭建Elasticsearch & Kibana
• • 安装docker
  • 创建配置文件
  • 执行步骤
• Filebeat配置

搭建Elasticsearch & Kibana 安装docker

需要安装docker 和docker compose ， 这部分的安装，不再赘述， 可以参考我之前的文章。

创建配置文件

创建instances.yml , 这里给es和kibana生成证书，也可以添加一个client，给filebeat用，不过我这里filebeat使用es的证书， 所以就没有创建client 的证书

instances:
  - name: es01
    dns:
      - es01
      - localhost
    ip:
      - 127.0.0.1

  - name: kib01
    dns:
      - kib01
      - localhost

配置 .env

COMPOSE_PROJECT_NAME=es
CERTS_DIR=/usr/share/elasticsearch/config/certificates
VERSION=7.8.0

创建 create-certs.yml ， 准备制造证书

version: '2.2'
 
services:
  create_certs:
    image: elasticsearch:${VERSION}
    container_name: create_certs
    command: >
      bash -c '
        yum install -y -q -e 0 unzip;
        if [[ ! -f /certs/bundle.zip ]]; then
          bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
          unzip /certs/bundle.zip -d /certs;
        fi;
        chown -R 1000:0 /certs
      '
    working_dir: /usr/share/elasticsearch
    volumes:
      - certs:/certs
      - .:/usr/share/elasticsearch/config/certificates
    networks:
      - elastic
 
volumes:
  certs:
    driver: local
 
networks:
  elastic:
    driver: bridge

接下来， 创建elastic-docker-tls.yml， 给es和kibana配置相关信息

version: '2.2'
 
services:
  es01:
    image: elasticsearch:${VERSION}
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    ports:
      - 9200:9200
    networks:
      - elastic
 
    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  kib01:
    image: kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: CHANGEME
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
    volumes:
      - certs:$CERTS_DIR
    networks:
      - elastic
volumes:
  data01:
    driver: local
  certs:
    driver: local
 
networks:
  elastic:
    driver: bridge

执行步骤

1. 先执行如下命令， 注意只执行1次就好
   docker-compose -f create-certs.yml run --rm create_certs

2. 执行完毕后， 可以检查如下目录是否生成证书相关信息

[root@es-jira volumes]# pwd
/var/lib/docker/volumes
[root@es-jira volumes]# ll
total 24
brw------- 1 root root 253, 1 Nov 17 21:45 backingFsBlockDev
drwxr-xr-x 3 root root     19 Nov 17 22:23 es_certs  #####注意是这个目录里面
drwx-----x 3 root root     19 Nov 17 22:28 es_data01
-rw------- 1 root root  32768 Nov 17 22:28 metadata.db

[root@es-jira _data]# ls -al /var/lib/docker/volumes/es_certs/_data
total 8
drwxr-xr-x 5 1000 root   59 Nov 17 22:27 .
drwxr-xr-x 3 root root   19 Nov 17 22:23 ..
-rw------- 1 1000 root 6182 Nov 17 22:27 bundle.zip
drwxrwxr-x 2 1000 root   20 Nov 17 22:27 ca
drwxrwxr-x 2 1000 root   38 Nov 17 22:27 es01
drwxrwxr-x 2 1000 root   40 Nov 17 22:27 kib01

3. 检查es和kib是否正确启动

[root@es-jira ~]# docker ps -a
ConTAINER ID   IMAGE                 COMMAND                  CREATED      STATUS                PORTS                              NAMES
51fcc32b62f4   kibana:7.8.0          "/usr/local/bin/dumb…"   2 days ago   Up 2 days             0.0.0.0:5601->5601/tcp             kib01
26bac6de7fd4   elasticsearch:7.8.0   "/tini -- /usr/local…"   2 days ago   Up 2 days (healthy)   0.0.0.0:9200->9200/tcp, 9300/tcp   es01

4. 集群启动之后， 生成默认用户的密码

docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords 
auto --batch --url https://es01:9200"

此处注意保留生成的密码， 我们还要调整下yml配置文件

5. 修改elastic-docker-tls.yml文件中 kibana_system 的密码

kib01:
    image: kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: CHANGEME   >>>>>>>>>>>>>>>>>>>>>>>>>>>此处
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt

6. 重启es 和kib

docker-compose -f elastic-docker-tls.yml stop
docker-compose -f elastic-docker-tls.yml up -d

7. 验证登录kib
   https://HOSTIP:5601
   

Filebeat配置

Filebeat的安装， 可以采用多种方式， 二进制或者docker方式都可以，这里我选用tar包安装方式。
重点是filebeat的配置，需要开启ssl及配置证书相关，截取部分配置如下：

output.elasticsearch:
    hosts: ["https://127.0.0.1:9200"]    ### 注意这里， 是127.0.0.1 ，如果你是其他地址， 在创建证书的时候就要考虑
    protocol: "https"
    username: elastic
    password: rMesfHfEETESJEliJSIv
    ssl.certificate_authorities: /root/filebeat/ca.crt
    ssl.certificate: /root/filebeat/es01.crt
    ssl.key: /root/filebeat/es01.key
    index: "tiktok-%{+yyyy.MM.dd}"