logstash过滤器：让所有的数组元素为新事件

Question

我试图让XML解析器后数组元素如下：

filter { 
    xml { 
     source => "message" 
     target => "xmldata" 
     store_xml => "false" 
     xpath => ["/OMA/ESMLog/LogEntry/Index/text()","index"] 
     xpath => ["/OMA/ESMLog/LogEntry/Status/text()","status"] 
     xpath => ["/OMA/ESMLog/LogEntry/TimeStampRaw/text()","timestampraw"] 
     xpath => ["/OMA/ESMLog/LogEntry/Description/text()","description"] 
    } 
     mutate { 
      remove_field => [ "message", "inxml", "xmldata" ] 
     } 

    mutate { 
    replace => { 
      "index" => "%{[index][0]}" 
     "status" => "%{[status][0]}" 
      "timestampraw" => "%{[timestampraw][0]}" 
      "description" => "%{[description][0]}" 

    } 
    } 
    date { 
     match => [ "timestampraw", "UNIX" ] 
    } 
} 

正如你可以看到我能够从阵列中获得每一个第一要素，但是我怎样才能从数组中获取所有元素作为新事件？ 因此，我希望将每个'LogEntry'元素看作XML中的新事件。 这里一些示例XML（原始XML来自OMSA）：

<?xml version="1.0" encoding="UTF-8"?> 
<OMA> 
<ESMLog> 
    <LogEntry> 
     <Index>0</Index> 
     <Status>2</Status> 
     <TimeStamp>Tue Nov 3 07:22:57 2015</TimeStamp> 
     <TimeStampRaw>1446535377</TimeStampRaw> 
     <Description>The system board Mem2 temperature is within range.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>1</Index> 
     <Status>3</Status> 
     <TimeStamp>System Boot</TimeStamp> 
     <TimeStampRaw>1446535378</TimeStampRaw> 
     <Description>The system board Mem2 temperature is less than the lower warning threshold.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>2</Index> 
     <Status>2</Status> 
     <TimeStamp>Mon Nov 2 14:17:09 2015</TimeStamp> 
     <TimeStampRaw>1446473829</TimeStampRaw> 
     <Description>Drive 0 is installed in disk drive bay 1.</Description> 
     </LogEntry> 
     <LogEntry> 
     <Index>3</Index> 
     <Status>4</Status> 
     <TimeStamp>Mon Nov 2 14:17:04 2015</TimeStamp> 
     <TimeStampRaw>1446473824</TimeStampRaw> 
     <Description>Drive 0 is removed from disk drive bay 1.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>4</Index> 
     <Status>2</Status> 
     <TimeStamp>Mon Nov 2 14:15:54 2015</TimeStamp> 
     <TimeStampRaw>1446473754</TimeStampRaw> 
     <Description>Drive 0 is installed in disk drive bay 1.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>5</Index> 
     <Status>4</Status> 
     <TimeStamp>Mon Nov 2 13:58:54 2015</TimeStamp> 
     <TimeStampRaw>1446472734</TimeStampRaw> 
     <Description>Drive 0 is removed from disk drive bay 1.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>6</Index> 
     <Status>2</Status> 
     <TimeStamp>Fri Feb 5 11:07:27 2010</TimeStamp> 
     <TimeStampRaw>1265368047</TimeStampRaw> 
     <Description>Drive 0 is installed in disk drive bay 1.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>7</Index> 
     <Status>2</Status> 
     <TimeStamp>Fri Feb 5 11:07:08 2010</TimeStamp> 
     <TimeStampRaw>1265368028</TimeStampRaw> 
     <Description>Drive 0 in disk drive bay 1 is operating normally.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>8</Index> 
     <Status>4</Status> 
     <TimeStamp>Fri Feb 5 11:07:07 2010</TimeStamp> 
     <TimeStampRaw>1265368027</TimeStampRaw> 
     <Description>Drive 0 is removed from disk drive bay 1.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>9</Index> 
     <Status>4</Status> 
     <TimeStamp>Fri Jan 29 09:33:27 2010</TimeStamp> 
     <TimeStampRaw>1264757607</TimeStampRaw> 
     <Description>Fault detected on drive 0 in disk drive bay 1.</Description> 
    </LogEntry> 
    <LogEntry> 
     <Index>10</Index> 
     <Status>2</Status> 
     <TimeStamp>Mon Feb 25 16:14:15 2008</TimeStamp> 
     <TimeStampRaw>1203956055</TimeStampRaw> 
     <Description>Log cleared.</Description> 
    </LogEntry> 
    <NumRecords>11</NumRecords> 
</ESMLog> 
<ObjStatus>2</ObjStatus> 
<SMStatus>0</SMStatus> 
</OMA> 

这里是我做出的解决方案，通过Jettro的例子：

filter { 
     xml { 
       source => "message" 
       target => "xmldata" 
       store_xml => "false" 
       xpath => ["/OMA/ESMLog//LogEntry","logentry"] 
     } 

     mutate { 
      remove_field => [ "message", "inxml", "xmldata" ] 
     } 

     split { 
       field => "[logentry]" 
     } 

     xml { 
       source => "logentry" 
       store_xml => "false" 
       xpath => ["/LogEntry/Index/text()","index"] 
       xpath => ["/LogEntry/Status/text()","status"] 
       xpath => ["/LogEntry/TimeStampRaw/text()","timestampraw"] 
       xpath => ["/LogEntry/Description/text()","description"] 
     } 
     mutate { 
       replace => { 
       "index" => "%{[index][0]}" 
       "status" => "%{[status][0]}" 
       "timestampraw" => "%{[timestampraw][0]}" 
       "description" => "%{[description][0]}" 

       } 
     } 
    date { 
      match => [ "timestampraw", "UNIX" ] 
    } 
      mutate { 
     remove_field => [ "logentry" , "timestampraw" ] 
    } 
} 

看来拆分后开始创建“循环“并处理来自更深层部分的所有数组。 谢谢

Answer 1

由于你的例子有点冗长，我尝试了一个更简单的xml，但是你应该能够摆脱你需要的东西。诀窍是使用分离滤波器。在我使用的配置下面和输出。

# <result><logline><description>item 1</description></logline><logline><description>item 2</description></logline></result> 
input { 
    stdin{} 
} 
filter { 
    xml { 
     source => "message" 
     store_xml => "false" 
     xpath => ["/result/logline","loglines"] 
     remove_field => [ "message", "host" ] 
    } 
    split { 
     field => "loglines" 
    } 
    xml { 
     source => "loglines" 
     store_xml => "false" 
     xpath => ["/logline/description/text()","description"] 
     remove_field => [ "loglines" ] 
    }  
} 
output { 
    stdout{ codec => rubydebug } 
} 

和输出就变成了：

{ 
     "@version" => "1", 
    "@timestamp" => "2016-06-07T09:40:35.420Z", 
      "host" => "Jettros-MBP.fritz.box", 
    "description" => [ 
     [0] "item 1" 
    ] 
} 
{ 
     "@version" => "1", 
    "@timestamp" => "2016-06-07T09:40:35.420Z", 
      "host" => "Jettros-MBP.fritz.box", 
    "description" => [ 
     [0] "item 2" 
    ] 
} 

正如你可以看到，现在有两个事件。