EKL日志分析平台-kibana数据可视化

EKL日志分析平台-kibana数据可视化

目录

简介  

实验环境

安装kibana

 可视化

内部信息查询

 启用xpack验证

连接elastic

head连接

优化

metricbeat 下载

filebeatElasticsearch module | Filebeat Reference [7.6] | Elastichttps://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-elasticsearch.html

---

简介  

       kibana 是一款开源的数据分析和可视化平台，它是 Elastic Stack 成员之一，设计用于和 Elasticsearch 协作。您可以使用 Kibana 对 Elasticsearch 索引中的数据进行搜索、查看、交互 *** 作。您可以很方便的利用图表、表格及地图对数据进行多元化的分析和呈现。

实验环境

server1-3 es集群优化好

安装kibana

准备安装包kibana-7.6.1-x86_64.rpm,server4

rpm -ivh kibana-7.6.1-x86_64.rpm 

编辑配置文件

cd /etc/kibana/

 115    i18n.locale: "zh-CN"  中文

37   kibana.index: ".kibana"

28  elasticsearch.hosts: ["http://172.25.7.1:9200"]

  2  server.port: 5601

 7   server.host: "172.25.7.4"

 启动服务查看端口

systemctl start kibana.service

 可视化

http://172.25.7.4:5601

 使用样例模板

选swap

左下脚manager进入管理，创建索引

 

 索引可视化v

 

 当我们再次收集apache

 server4 logstash -f /etc/logstash/conf.d/test.conf


主机
 ab -c1 -n 500 http://172.25.7.4/index.html

创建可视化垂直

加xy轴

server1主机访问步骤一样

 保存，top3

创建仪表板dashbord

 

 

 

 保存

刷新时间

内部信息查询

 启用xpack验证

server1

找到证书工工具

[root@server1 bin]# cd ..
[root@server1 elasticsearch]# bin/elasticsearch-certutil ca

 回车

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

[root@server1 elasticsearch]# ls
bin                       jdk          modules     README.asciidoc
elastic-certificates.p12  lib          NOTICE.txt
elastic-stack-ca.p12      LICENSE.txt  plugins
[root@server1 elasticsearch]

cp elastic-certificates.p12 /etc/elasticsearch/

  每个节点的都需要

chown elasticsearch elastic-certificates.p12

vim elasticsearch.yml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

证书复制

 scp -p elastic-certificates.p12 server2:/etc/elasticsearch/elastic-certificates.p12

 重启服务

连接elastic

cd /usr/share/elasticsearch/bin/

[root@server1 bin]# ls
elasticsearch                elasticsearch-saml-metadata
elasticsearch-certgen        elasticsearch-setup-passwords
elasticsearch-certutil       elasticsearch-shard
elasticsearch-cli            elasticsearch-sql-cli
elasticsearch-croneval       elasticsearch-sql-cli-7.6.1.jar
elasticsearch-env            elasticsearch-syskeygen
elasticsearch-env-from-file  elasticsearch-users
elasticsearch-keystore       x-pack-env
elasticsearch-migrate        x-pack-security-env
elasticsearch-node           x-pack-watcher-env
elasticsearch-plugin
[root@server1 bin]#

 ./elasticsearch-setup-passwords --help
./elasticsearch-setup-passwords interactive
交互式设密码

server4

[root@server4 conf.d]# cd /etc/kibana/
[root@server4 kibana]# ls
apm.js  kibana.yml
[root@server4 kibana]# vim kibana.yml

elasticsearch.username: "kibana"
elasticsearch.password: "westos"

 systemctl restart kibana.service

 自己在交互式式的密码

连接logstash

[root@server4 conf.d]# vim test.conf
[root@server4 conf.d]# cat test.conf

input {
    file {
       path => "/var/log/httpd/access_log"
        start_position => "beginning"
    }

}

filter {
    grok {
    match => { "message" => "%{HTTPD_COMBINEDLOG}" }
    }
}

output {
    stdout {}

    elasticsearch {
        hosts => ["172.25.7.1:9200"]
        index => "apachelog-%{+yyyy.MM.dd}"
                user => "elastic"
                password => "westos"
    }
    
}

logstash -f test.conf

head连接

[westos@students72 Desktop]$ podman ps
ConTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
[westos@students72 Desktop]$ podman ps -a
ConTAINER ID  IMAGE                                COMMAND               CREATED       STATUS   PORTS                   NAMES
dcb45a704ec0  docker.io/mobz/elasticsearch-head:5  /bin/sh -c grunt ...  28 hours ago  Created  0.0.0.0:9100->9100/tcp  head
a22dbf1d5e7f  docker.io/lmenezes/cerebro:latest                          28 hours ago  Created  0.0.0.0:9000->9000/tcp  cerbro
[westos@students72 Desktop]$ podman start cerbro
cerbro
[westos@students72 Desktop]$

 

Head连接

 2.server

vim /etc/elasticsearch/elasticsearch.yml

添加

http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

重启服务

 http://172.25.7.1:9100//?auth_user=elastic&auth_password=westos

访问连接

优化

metricbeat 下载

 rpm -ivh metricbeat-7.6.1-x86_64.rpm

metricbeat modules enable elasticsearch-xpack

 

 vim elasticsearch-xpack.yml

[root@server1 metricbeat]# vim metricbeat.yml
 systemctl start metricbeat.service

 

 内部采集进入集群节点3

 2.3同样安装

filebeatElasticsearch module | Filebeat Reference [7.6] | Elastichttps://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-elasticsearch.html

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-elasticsearch.html

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /var/log/elasticsearch/*.log       
      - /var/log/elasticsearch/*_server.json

  gc:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /var/log/elasticsearch/gc.log.[0-9]*
      - /var/log/elasticsearch/gc.log

  audit:
    enabled: true
    # Set custom paths for the log files. If left empty,

  # Filebeat will choose the paths depending on your OS.
     var.paths:
       - /var/log/elasticsearch/*_access.log
       - /var/log/elasticsearch/*_audit.json

   slowlog:
     enabled: true
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     var.paths:
       - /var/log/elasticsearch/*_index_search_slowlog.log 
       - /var/log/elasticsearch/*_index_indexing_slowlog.log
       - /var/log/elasticsearch/*_index_search_slowlog.json
       - /var/log/elasticsearch/*_index_indexing_slowlog.json

   deprecation:
     enabled: true
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     var.paths:
       - /var/log/elasticsearch/*_deprecation.log
       - /var/log/elasticsearch/*_deprecation.json

vim filebeat.yml根metricbeat一样

重启服务