使用CSV对Logstash进行分析和计算

我在解析和计算性能时遇到问题导航计时数据我在csv中。使用CSV对Logstash进行分析和计算

我能够解析字段，但不知道如何正确地进行计算（下面）。有几点要牢记：

数据集由加粗值组合在一起（这是在21个数据点被带到 ACMEPage-1486643427973，unloadEventEnd，1486643372422个 2.Calculations需要的TS要与数据进行积分我假设需要做一些标记和分组，但是我对如何实现它没有清晰的认识，任何帮助都将不胜感激。

感谢，

---------------计算-----------------

总第一个字节时间= responseStart - navigationStart

时延= responseStart - fetchStart

DNS /域名查找时间= domainLookupEnd - domainLookupStart

服务器连接时间= connectEnd - connectStart

服务器响应时间= responseStart - requestStart

页面加载时间= loadEventStart - navigationStart

传输/网页下载时间= responseEnd - responseStart

DOM互动时间= domInteractive - navigationStart

DOM内容加载时间= domContentLoadedEventEnd - navigationStart

DOM Processing to Interactive = domInteractive - domLoading

DOM Interactive to Complete = domComplete - domInteractive

的Onload = loadEventEnd - loadEventStart

-------数据在CSV -----------

ACMEPage-1486643427973，unloadEventEnd，1486643372422 ACMEPage-1486643427973，responseEnd，1486643372533 ACMEPage-1486643427973，responseStart，1486643372416 ACMEPage-1486643427973，domInteractive，1486643373030 ACMEPage-1486643427973，domainLookupEnd，1486643372194 ACMEPage-1486643427973，卸载EventStart，1486643372422 ACMEPage-1486643427973，domComplete，1486643373512 ACMEPage-1486643427973，domContentLoadedEventStart，1486643373030 ACMEPage-1486643427973，domainLookupStart，1486643372194 ACMEPage-1486643427973，redirectEnd，0 ACMEPage-1486643427973，redirectStart，0 ACMEPage-1486643427973，connectEnd， 1486643372194 ACMEPage-1486643427973，的toJSON，{} ACMEPage-1486643427973，connectStart，1486643372194 ACMEPage-1486643427973，loadEventStart，1486643373512 ACMEPage-1486643427973，navigationStart，1486643372193 ACMEPage-1486643427973，requestStart，1486643372203 ACMEPage-1486643427973，secureConnectionStart，0 ACMEPage-1486643427973，fetchStart，1486643372194 ACMEPage-1486643427973，domContentLoadedEventEnd，1486643373058 ACMEPage-1486643427973，domLoading，1486643372433 ACMEPage-1486643427973，loadEventEnd，1486643373514

----------输出 - --------------

"path" => "/Users/philipp/Downloads/build2/logDataPoints_com.concur.automation.cge.ui.admin.ADCLookup_1486643340910.csv", 
"@timestamp" => 2017-02-09T12:29:57.763Z, 
"navigationTimer" => "connectStart", 
"@version" => "1", 
"host" => "15mbp-09796.local", 
"elapsed_time" => "1486643372194", 
"pid" => "1486643397763", 
"page" => "ADCLookupDataPage", 
"message" => "ADCLookupDataPage-1486643397763,connectStart,1486643372194", 
"type" => "csv" 
}

-------------- logstash。CONF ----------------

input { 
file { 
type => "csv" 
path => "/Users/path/logDataPoints_com.concur.automation.acme.ui.admin.acme_1486643340910.csv" 
start_position => beginning 
# to read from the beginning of file 
sincedb_path => "/dev/null" 
} 
} 

filter { 
csv { 
columns => ["page_id", "navigationTimer", "elapsed_time"] 
} 

if (["elapsed_time"] == "{}") { 
drop{} 
} 
else { 
grok { 
match => { "page_id" => "%{WORD:page}-%{INT:pid}" 
} 

remove_field => [ "page_id" ] 
} 
} 

date { 
match => [ "pid", "UNIX_MS" ] 
target => "@timestamp" 
} 
} 

output { 
elasticsearch { hosts => ["localhost:9200"] } 
stdout { codec => rubydebug } 
}

来源

2017-02-13 ppragados

我下面让我的趋势数据：

-I发现更容易转动的数据，而不是去往下通过塔，为具有数据沿着行去每每个“事件”或“文档”

需要被相应地映射为一个整数或串

一旦数据是 - 每个字段正确的Kibana我有使用蹭的问题y代码过滤器来进行简单的数学计算，因此我最终使用“脚本字段”在Kibana中进行了计算。

input { 
    file { 
     type => "csv" 
    path => "/Users/philipp/perf_csv_pivot2.csv" 
    start_position => beginning 
    # to read from the beginning of file 
    sincedb_path => "/dev/null" 
    } 
} 


filter { 
    csv { 
     columns => ["page_id","unloadEventEnd","responseEnd","responseStart","domInteractive","domainLookupEnd","unloadEventStart","domComplete","domContentLoadedEventStart","domainLookupstart","redirectEnd","redirectStart","connectEnd","toJSON","connectStart","loadEventStart","navigationStart","requestStart","secureConnectionStart","fetchStart","domContentLoadedEventEnd","domLoading","loadEventEnd"] 
    }

神交{ 匹配=> { “PAGE_ID”=> “％{WORD：页} - ％{INT：page_ts}”} remove_field => [ “PAGE_ID”， “消息”，“路径“] }

mutate { 
    convert => { "unloadEventEnd" => "integer" } 
    convert => { "responseEnd" => "integer" } 
    convert => { "responseStart" => "integer" } 
    convert => { "domInteractive" => "integer" } 
    convert => { "domainLookupEnd" => "integer" } 
    convert => { "unloadEventStart" => "integer" } 
    convert => { "domComplete" => "integer" } 
    convert => { "domContentLoadedEventStart" => "integer" } 
    convert => { "domainLookupstart" => "integer" } 
    convert => { "redirectEnd" => "integer" } 
    convert => { "redirectStart" => "integer" } 
    convert => { "connectEnd" => "integer" } 
    convert => { "toJSON" => "string" } 
    convert => { "connectStart" => "integer" } 
    convert => { "loadEventStart" => "integer" } 
    convert => { "navigationStart" => "integer" } 
    convert => { "requestStart" => "integer" } 
    convert => { "secureConnectionStart" => "integer" } 
    convert => { "fetchStart" => "integer" } 
    convert => { "domContentLoadedEventEnd" => "integer" } 
    convert => { "domLoading" => "integer" } 
    convert => { "loadEventEnd" => "integer" } 

    } 


    date { 
     match => [ "page_ts", "UNIX_MS" ] 
     target => "@timestamp" 
     remove_field => [ "page_ts", "timestamp", "host", "toJSON" ] 
     } 
    } 
output { 
    elasticsearch { hosts => ["localhost:9200"] } 
    stdout { codec => rubydebug } 
    }

希望这可以帮助别人，

来源

2017-02-15 00:10:22 ppragados

使用CSV对Logstash进行分析和计算

回答

相关问题