删除注册表时重复的日志事件

Question

我目前正在进行PoC ELK安装，并且想重新发送在Filebeat中注册的文件的每个日志行以用于测试目的。

这是我做的：

1. 我停止Filebeat
2. 我删除Logstash指数通过Kibana
3. 我删除Filebeat注册表文件
4. 我开始Filebeat

在Kibana中，我可以看到日志行数是事件的两倍，我也可以看到每个事件都被复制过一次。

这是为什么？

Filebeat日志：

2017-05-05T14:25:16+02:00 INFO Setup Beat: filebeat; Version: 5.2.2 
2017-05-05T14:25:16+02:00 INFO Max Retries set to: 3 
2017-05-05T14:25:16+02:00 INFO Activated logstash as output plugin. 
2017-05-05T14:25:16+02:00 INFO Publisher name: anonymized 
2017-05-05T14:25:16+02:00 INFO Flush Interval set to: 1s 
2017-05-05T14:25:16+02:00 INFO Max Bulk Size set to: 2048 
2017-05-05T14:25:16+02:00 INFO filebeat start running. 
2017-05-05T14:25:16+02:00 INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file. 
2017-05-05T14:25:16+02:00 INFO Loading registrar data from /var/lib/filebeat/registry 
2017-05-05T14:25:16+02:00 INFO States Loaded from registrar: 0 
2017-05-05T14:25:16+02:00 INFO Loading Prospectors: 1 
2017-05-05T14:25:16+02:00 INFO Prospector with previous states loaded: 0 
2017-05-05T14:25:16+02:00 INFO Loading Prospectors completed. Number of prospectors: 1 
2017-05-05T14:25:16+02:00 INFO All prospectors are initialised and running with 0 states to persist 
2017-05-05T14:25:16+02:00 INFO Starting Registrar 
2017-05-05T14:25:16+02:00 INFO Start sending events to output 
2017-05-05T14:25:16+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s 
2017-05-05T14:25:16+02:00 INFO Starting prospector of type: log 
2017-05-05T14:25:16+02:00 INFO Harvester started for file: /some/where/anonymized.log 
2017-05-05T14:25:46+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=2 libbeat.logstash.publish.read_bytes=54 libbeat.logstash.publish.write_bytes=32390 libbeat.logstash.published_and_acked_events=578 filebeat.harvester.running=1 registar.states.current=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.publisher.published_events=578 publish.events=579 filebeat.harvester.started=1 registrar.states.update=579 filebeat.harvester.open_files=1 
2017-05-05T14:26:16+02:00 INFO No non-zero metrics in the last 30s 

Answer 1

删除注册表文件创建的问题。

Filebeat管理文件的状态以及探测器（在内存中）和注册表文件（保存在磁盘中）的事件ACK。

请阅读文档Here

您可以管理自己的每个事件的_id领域，使得被复制（以任何理由，即使在生产环境）任何情况下将不会有他们两个在elasticsearch ，但会更新事件。

在logstash管道配置文件中创建以下配置。

#if your logs don't have a unique ID, use the following to generate one 
fingerprint{ 
     #with the message field or choose other(s) that can give you a uniqueID 
     source => ["message"] 
     target => "LogID" 
     key => "something" 
     method => "MD5" 
     concatenate_sources => true 
      } 
#in your output section 
    elasticsearch{ 
        hosts => ["localhost:9200"] 
        document_id => "%{LogID}" 
        index => "yourindex" 
     }