如何解析/过滤器日志藏匿配置有多个时间戳和线

Question

我有以下的日志文件

2017-06-13 13:00:01,494 - INFO [Line: 48]: Begin logging 

2017-06-13 13:00:01,494 - DEBUG [Line: 89]: Writing to lockfile. Lockfile location: /etc/conf/the/back_up.txt 

2017-06-13 13:00:03,521 - WARNING [Line: 449]: Snapshotting is not enabled 

2017-06-13 13:06:15,663 - INFO [Line: 898]: stderr: 

17/06/13 13:00:13 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=true,maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/DAT/ABC], targetPath=/etc/conf/the/back_up.txt, tar getPathExists=true, preserveRawXattrs=false} 

17/06/13 13:00:13 INFO impl.TimelineClientImpl: Timeline service address: http://ip-192-168-X-XX.xyz:9000/v1/example/ 
17/06/13 13:00:14 INFO tools.DistCp: DistCp job log path: /var/tar/xar 

17/06/13 13:00:20 INFO mapreduce.JobSubmitter: number of splits:22 

17/06/13 13:00:21 INFO impl.YarnClientImpl: Submitted application application_1495940390018_0989 

17/06/13 13:00:21 INFO mapreduce.Job: Running job: job_1495940390018_0989 
17/06/13 13:00:29 INFO mapreduce.Job: Job job_1495940390018_0989 running in uber mode : false 
17/06/13 13:00:29 INFO mapreduce.Job: map 0% reduce 0% 
17/06/13 13:00:46 INFO mapreduce.Job: map 11% reduce 0% 
17/06/13 13:00:47 INFO mapreduce.Job: map 17% reduce 0% 
17/06/13 13:00:48 INFO mapreduce.Job: map 18% reduce 0% 
17/06/13 13:00:49 INFO mapreduce.Job: map 23% reduce 0% 
17/06/13 13:00:50 INFO mapreduce.Job: map 28% reduce 0% 
17/06/13 13:00:51 INFO mapreduce.Job: map 29% reduce 0% 
17/06/13 13:00:52 INFO mapreduce.Job: map 32% reduce 0% 
17/06/13 13:00:53 INFO mapreduce.Job: map 37% reduce 0% 
17/06/13 13:00:54 INFO mapreduce.Job: map 38% reduce 0% 
17/06/13 13:00:55 INFO mapreduce.Job: map 41% reduce 0% 
17/06/13 13:00:56 INFO mapreduce.Job: map 44% reduce 0% 
17/06/13 13:00:57 INFO mapreduce.Job: map 45% reduce 0% 
17/06/13 13:00:58 INFO mapreduce.Job: map 47% reduce 0% 
17/06/13 13:00:59 INFO mapreduce.Job: map 48% reduce 0% 
17/06/13 13:01:00 INFO mapreduce.Job: map 49% reduce 0% 
17/06/13 13:01:07 INFO mapreduce.Job: map 54% reduce 0% 
17/06/13 13:01:08 INFO mapreduce.Job: map 57% reduce 0% 
17/06/13 13:01:10 INFO mapreduce.Job: map 59% reduce 0% 
17/06/13 13:01:11 INFO mapreduce.Job: map 60% reduce 0% 
17/06/13 13:01:13 INFO mapreduce.Job: map 62% reduce 0% 
17/06/13 13:01:14 INFO mapreduce.Job: map 63% reduce 0% 
17/06/13 13:01:15 INFO mapreduce.Job: map 64% reduce 0% 
17/06/13 13:01:16 INFO mapreduce.Job: map 65% reduce 0% 
17/06/13 13:01:31 INFO mapreduce.Job: map 76% reduce 0% 
17/06/13 13:01:35 INFO mapreduce.Job: map 77% reduce 0% 
17/06/13 13:01:39 INFO mapreduce.Job: map 78% reduce 0% 
17/06/13 13:01:44 INFO mapreduce.Job: map 79% reduce 0% 
17/06/13 13:01:48 INFO mapreduce.Job: map 80% reduce 0% 
17/06/13 13:01:52 INFO mapreduce.Job: map 81% reduce 0% 
17/06/13 13:01:55 INFO mapreduce.Job: map 82% reduce 0% 
17/06/13 13:01:58 INFO mapreduce.Job: map 83% reduce 0% 
17/06/13 13:02:01 INFO mapreduce.Job: map 84% reduce 0% 
17/06/13 13:02:06 INFO mapreduce.Job: map 85% reduce 0% 
17/06/13 13:02:09 INFO mapreduce.Job: map 86% reduce 0% 
17/06/13 13:02:12 INFO mapreduce.Job: map 87% reduce 0% 
17/06/13 13:02:16 INFO mapreduce.Job: map 88% reduce 0% 
17/06/13 13:02:18 INFO mapreduce.Job: map 89% reduce 0% 
17/06/13 13:02:23 INFO mapreduce.Job: map 90% reduce 0% 
17/06/13 13:02:28 INFO mapreduce.Job: map 91% reduce 0% 
17/06/13 13:02:36 INFO mapreduce.Job: map 92% reduce 0% 
17/06/13 13:02:42 INFO mapreduce.Job: map 93% reduce 0% 
17/06/13 13:02:47 INFO mapreduce.Job: map 94% reduce 0% 
17/06/13 13:02:51 INFO mapreduce.Job: map 95% reduce 0% 
17/06/13 13:02:57 INFO mapreduce.Job: map 96% reduce 0% 
17/06/13 13:03:04 INFO mapreduce.Job: map 97% reduce 0% 
17/06/13 13:03:10 INFO mapreduce.Job: map 98% reduce 0% 
17/06/13 13:03:30 INFO mapreduce.Job: map 99% reduce 0% 
17/06/13 13:03:58 INFO mapreduce.Job: map 100% reduce 0% 

17/06/13 13:06:15 INFO mapreduce.Job: Job job_1495940390018_0989 completed successfully 

17/06/13 13:06:15 INFO mapreduce.Job: Counters: 33 
File System Counters 
FILE: Number of bytes read=0 
FILE: Number of bytes written=30634 
FILE: Number of read operations=0 
FILE: Number of large read operations=0 
FILE: Number of write operations=0 
HDFS: Number of bytes read=1810172 
HDFS: Number of bytes written=6602 
HDFS: Number of read operations=21710 
HDFS: Number of large read operations=0 
HDFS: Number of write operations=4461 
Job Counters 
Launched map tasks=22 
Other local map tasks=22 
Total time spent by all maps in occupied slots (ms)=09878 
Total time spent by all reduces in occupied slots (ms)=0 
Total time spent by all map tasks (ms)=170939 
Total vcore-milliseconds taken by all map tasks=17049 
Total megabyte-milliseconds taken by all map tasks=1747536 
Map-Reduce Framework 
Map input records=417 
Map output records=175 
Input split bytes=262 
Spilled Records=0 
Failed Shuffles=0 
Merged Map outputs=0 
GC time elapsed (ms)=3338 
CPU time spent (ms)=3180 
Physical memory (bytes) snapshot=480768 
Virtual memory (bytes) snapshot=61798624 
Total committed heap usage (bytes)=2965728 
File Input Format Counters 
Bytes Read=17510 
File Output Format Counters 
Bytes Written=6616 
org.apache.hadoop.tools.mapred.CopyMapper$Counter 
BYTESSKIPPED=11361 
COPY=1242 
SKIP=3175 
2017-06-13 13:06:15,668 - INFO [Line: 904]: Distcp -log output stored in /var/AB/CY/ 

2017-06-13 13:06:15,673 - INFO [Line: 132]: End logging 

2017-06-13 13:07:01,494 - INFO [Line: 48]: Begin logging 

. <similar to above logs> 

. <similar to above logs> 

. <similar to above logs> 

2017-06-13 13:07:15,673 - INFO [Line: 132]: End logging 

.. 这个复杂的日志文件。

。 。 。

等

再然后就会有类似于开始“开始记录”不同的作业ID上面的部分，当你在上面加粗看到“结束记录”结束。

所以我在这里的问题是：如何解析日志存储配置中的日志，我想在每个ES日志记录中看到每个块。我的意思是块是从“开始记录”到“结束记录”的所有内容 *注意：我对ES仍然陌生，所以任何建议都应该对我有用，因为我提前感谢！

Answer 1

您是否尝试过使用Logstash的multiline codec？
下面是关于存储多行活动多一些文档：Managing Multiline Events

我在你的情况想象你也许可以做这样的事情

multilineinput.config：

input { 
    stdin { 
    codec => multiline { 
     pattern => "End logging" 
     what => "next" 
     negate => true 
    } 
    } 
} 

output { 
    stdout { 
    codec => "rubydebug" 
    } 
} 

主要生产：

{ 
     "@version" => "1", 
     "message" => "2017-06-13 13:00:01,494 - INFO [Line: 48]: Begin logging\n\n2017-06-13 13:00:01,494 - DEBUG [Line: 89]: Writing to lockfile. Lockfile location: /etc/conf/the/back_up.txt\n\n2017-06-13 13:00:03,521 - WARNING [Line: 449]: Snapshotting is not enabled\n\n2017-06-13 13:06:15,663 - INFO [Line: 898]: stderr: \n\n17/06/13 13:00:13 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=true, deleteMissing=false, ignoreFailures=true,maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[/DAT/ABC], targetPath=/etc/conf/the/back_up.txt, tar getPathExists=true, preserveRawXattrs=false}\n\n17/06/13 13:00:13 INFO impl.TimelineClientImpl: Timeline service address: http://ip-192-168-X-XX.xyz:9000/v1/example/\n17/06/13 13:00:14 INFO tools.DistCp: DistCp job log path: /var/tar/xar\n\n17/06/13 13:00:20 INFO mapreduce.JobSubmitter: number of splits:22\n\n17/06/13 13:00:21 INFO impl.YarnClientImpl: Submitted application application_1495940390018_0989\n\n17/06/13 13:00:21 INFO mapreduce.Job: Running job: job_1495940390018_0989 \n17/06/13 13:00:29 INFO mapreduce.Job: Job job_1495940390018_0989 running in uber mode : false\n17/06/13 13:00:29 INFO mapreduce.Job: map 0% reduce 0%\n17/06/13 13:00:46 INFO mapreduce.Job: map 11% reduce 0%\n17/06/13 13:00:47 INFO mapreduce.Job: map 17% reduce 0%\n17/06/13 13:00:48 INFO mapreduce.Job: map 18% reduce 0%\n17/06/13 13:00:49 INFO mapreduce.Job: map 23% reduce 0%\n17/06/13 13:00:50 INFO mapreduce.Job: map 28% reduce 0%\n17/06/13 13:00:51 INFO mapreduce.Job: map 29% reduce 0%\n17/06/13 13:00:52 INFO mapreduce.Job: map 32% reduce 0%\n17/06/13 13:00:53 INFO mapreduce.Job: map 37% reduce 0%\n17/06/13 13:00:54 INFO mapreduce.Job: map 38% reduce 0%\n17/06/13 13:00:55 INFO mapreduce.Job: map 41% reduce 0%\n17/06/13 13:00:56 INFO mapreduce.Job: map 44% reduce 0%\n17/06/13 13:00:57 INFO mapreduce.Job: map 45% reduce 0%\n17/06/13 13:00:58 INFO mapreduce.Job: map 47% reduce 0%\n17/06/13 13:00:59 INFO mapreduce.Job: map 48% reduce 0%\n17/06/13 13:01:00 INFO mapreduce.Job: map 49% reduce 0%\n17/06/13 13:01:07 INFO mapreduce.Job: map 54% reduce 0%\n17/06/13 13:01:08 INFO mapreduce.Job: map 57% reduce 0%\n17/06/13 13:01:10 INFO mapreduce.Job: map 59% reduce 0%\n17/06/13 13:01:11 INFO mapreduce.Job: map 60% reduce 0%\n17/06/13 13:01:13 INFO mapreduce.Job: map 62% reduce 0%\n17/06/13 13:01:14 INFO mapreduce.Job: map 63% reduce 0%\n17/06/13 13:01:15 INFO mapreduce.Job: map 64% reduce 0%\n17/06/13 13:01:16 INFO mapreduce.Job: map 65% reduce 0%\n17/06/13 13:01:31 INFO mapreduce.Job: map 76% reduce 0%\n17/06/13 13:01:35 INFO mapreduce.Job: map 77% reduce 0%\n17/06/13 13:01:39 INFO mapreduce.Job: map 78% reduce 0%\n17/06/13 13:01:44 INFO mapreduce.Job: map 79% reduce 0%\n17/06/13 13:01:48 INFO mapreduce.Job: map 80% reduce 0%\n17/06/13 13:01:52 INFO mapreduce.Job: map 81% reduce 0%\n17/06/13 13:01:55 INFO mapreduce.Job: map 82% reduce 0%\n17/06/13 13:01:58 INFO mapreduce.Job: map 83% reduce 0%\n17/06/13 13:02:01 INFO mapreduce.Job: map 84% reduce 0%\n17/06/13 13:02:06 INFO mapreduce.Job: map 85% reduce 0%\n17/06/13 13:02:09 INFO mapreduce.Job: map 86% reduce 0%\n17/06/13 13:02:12 INFO mapreduce.Job: map 87% reduce 0%\n17/06/13 13:02:16 INFO mapreduce.Job: map 88% reduce 0%\n17/06/13 13:02:18 INFO mapreduce.Job: map 89% reduce 0%\n17/06/13 13:02:23 INFO mapreduce.Job: map 90% reduce 0%\n17/06/13 13:02:28 INFO mapreduce.Job: map 91% reduce 0%\n17/06/13 13:02:36 INFO mapreduce.Job: map 92% reduce 0%\n17/06/13 13:02:42 INFO mapreduce.Job: map 93% reduce 0%\n17/06/13 13:02:47 INFO mapreduce.Job: map 94% reduce 0%\n17/06/13 13:02:51 INFO mapreduce.Job: map 95% reduce 0%\n17/06/13 13:02:57 INFO mapreduce.Job: map 96% reduce 0%\n17/06/13 13:03:04 INFO mapreduce.Job: map 97% reduce 0%\n17/06/13 13:03:10 INFO mapreduce.Job: map 98% reduce 0%\n17/06/13 13:03:30 INFO mapreduce.Job: map 99% reduce 0%\n17/06/13 13:03:58 INFO mapreduce.Job: map 100% reduce 0%\n\n17/06/13 13:06:15 INFO mapreduce.Job: Job job_1495940390018_0989 completed successfully \n\n17/06/13 13:06:15 INFO mapreduce.Job: Counters: 33\nFile System Counters\nFILE: Number of bytes read=0\nFILE: Number of bytes written=30634\nFILE: Number of read operations=0\nFILE: Number of large read operations=0\nFILE: Number of write operations=0\nHDFS: Number of bytes read=1810172\nHDFS: Number of bytes written=6602\nHDFS: Number of read operations=21710\nHDFS: Number of large read operations=0\nHDFS: Number of write operations=4461\nJob Counters \nLaunched map tasks=22\nOther local map tasks=22\nTotal time spent by all maps in occupied slots (ms)=09878\nTotal time spent by all reduces in occupied slots (ms)=0\nTotal time spent by all map tasks (ms)=170939\nTotal vcore-milliseconds taken by all map tasks=17049\nTotal megabyte-milliseconds taken by all map tasks=1747536\nMap-Reduce Framework\nMap input records=417\nMap output records=175\nInput split bytes=262\nSpilled Records=0\nFailed Shuffles=0\nMerged Map outputs=0\nGC time elapsed (ms)=3338\nCPU time spent (ms)=3180\nPhysical memory (bytes) snapshot=480768\nVirtual memory (bytes) snapshot=61798624\nTotal committed heap usage (bytes)=2965728\nFile Input Format Counters \nBytes Read=17510\nFile Output Format Counters \nBytes Written=6616\norg.apache.hadoop.tools.mapred.CopyMapper$Counter\nBYTESSKIPPED=11361\nCOPY=1242\nSKIP=3175\n2017-06-13 13:06:15,668 - INFO [Line: 904]: Distcp -log output stored in /var/AB/CY/\n\n2017-06-13 13:06:15,673 - INFO [Line: 132]: End logging ", 
      "tags" => [ 
     [0] "multiline" 
    ] 
} 
{ 
     "@version" => "1", 
     "message" => "\n2017-06-13 13:07:01,494 - INFO [Line: 48]: Begin logging\n\n. <similar to above logs>\n\n. <similar to above logs>\n\n. <similar to above logs>\n\n2017-06-13 13:07:15,673 - INFO [Line: 132]: End logging", 
      "tags" => [ 
     [0] "multiline" 
    ] 
}