filebeat收集多类型日志并设置不同的索引

filebeat收集多类型日志并设置不同的索引

一. filebeat分别收集Nginx正常和错误日志

filebeat配置文件:

1）、filebeat配收集nginx、tomcat日志

filebeat.inputs:
############nginx##############
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

#############tomcat###############

- type: log
  enabled: true
  tags: ["tomcat"]
 
#############output###############

setup.kibana:
   host: "10.0.0.31:5601"

output.elasticsearch:
  hosts: ["10.0.0.31:9200"]
  #index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
  indices:
    - index: "nginx_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "access"
    - index: "nginx_error-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "error"
    - index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"
      when.contains:
        tags: "tomcat"

###########nginx模板###############
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
###########tomcat模板##############
setup.template.name: "tomcat"
setup.template.pattern: "tomcat-*"
setup.template.enabled: false
setup.template.overwrite: true

 2)、filebeat收集不同日志

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  fields:
    log_type: "nginx"
  json.key_under_root: true
  json.overwite_keys: true

- type: log
  enabled: true
  paths:
    - /var/log/elasticsearch/elasticsearch.log
  fields:
    log_type: "es"

  multiline.pattern: '^s'
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true
  paths:
    - /data/ruoyi/*.log
  fields:
    log_type: "ruoyi"

  multiline.pattern: '^s'
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["192.168.53.21:9200","192.168.53.22:9200"]
  index: "nginx-%{+yyyy.MM.dd}"
  indices:
    - index: "es-log"
      when.contains:
        fields:
          log_type: "es"
    - index: "ruoyi-log"
      when.contains:
        fields:
          log_type: "ruoyi"

###########nginx模板##############
setup.ilm.enabled: false                    # 如果要创建多个索引，需要将此项设置为 false
setup.template.name: web-nginx-log          # 设置模板的名称
setup.template.pattern: web-nginx-*         # 设置模板的匹配方式，上面索引的前缀要和这里保持一致
setup.template.overwrite: true
setup.template.enabled: false
###########es模板##############
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true
###########ruoyi模板##############
setup.template.name: "ruoyi"
setup.template.pattern: "ruoyi-*"
setup.template.enabled: false
setup.template.overwrite: true

编辑完成后，启动 filebeat 进程。到 Kibana 中查看索引列表，可以发现已经有三个新创建的索引