我们使用Ansible并拥有Logstash。测试自定义Logstash过滤器
如何编写一些测试来覆盖我们的自定义logstash过滤器?我想要做的是这样的:
- 给予Logstash配置有过滤器,
- 传一个日志行(或多行日志记录),
- 看到它成功地解析成片。
我知道这是 - https://github.com/elastic/logstash/wiki/Tips:Testing-your-filters,但我不明白这是如何有用 - 它似乎已过时。
我们使用Ansible并拥有Logstash。测试自定义Logstash过滤器
如何编写一些测试来覆盖我们的自定义logstash过滤器?我想要做的是这样的:
我知道这是 - https://github.com/elastic/logstash/wiki/Tips:Testing-your-filters,但我不明白这是如何有用 - 它似乎已过时。
回答行之有效高达logstash的1.5版本。此后,grok模式从核心中移除并放入logstash-core-patterns中。在logstash 2.2中对我有用的是:
# encoding: utf-8
require 'spec_helper'
require "logstash/patterns/core"
# solution based on https://github.com/logstash-plugins/logstash-filter-grok/blob/master/spec/filters/grok_spec.rb
module LogStash::Environment
# running the grok code outside a logstash package means
# LOGSTASH_HOME will not be defined, so let's set it here
# before requiring the grok filter
# the path that is set is the plugin root path
unless self.const_defined?(:LOGSTASH_HOME)
LOGSTASH_HOME = File.expand_path("../../../", __FILE__)
end
# also :pattern_path method must exist so we define it too
# method is called by logstash-filter-grok to create patterns_path array
#
# logstash-filter-grok/lib/logstash/filters/grok.rb(line ~230):
#
# @@patterns_path += [
# LogStash::Patterns::Core.path,
# LogStash::Environment.pattern_path("*")
#
# patterns defined in spec/patterns/ will be joined to the array by grok
unless self.method_defined?(:pattern_path)
def pattern_path(path)
::File.join(LOGSTASH_HOME, "spec", "patterns", path)
end
end
end
require "logstash/filters/grok"
require "logstash/filters/<tested-plugin>"
其余规范示例仍然有效。
随着新的依赖关系要求的到来,Gemfile也必须更改。我gemspec依赖关系是这样的:
# Gem dependencies
s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0"
s.add_runtime_dependency "<tested-plugin>"
s.add_development_dependency 'logstash-devutils', '~> 0'
s.add_development_dependency 'logstash-filter-grok', '~> 3.2'
s.add_development_dependency 'logstash-patterns-core', '~> 4.0'
一个完整的工作exapmle可以发现here
我发现this并结束了以下工作的测试代码:
# simple_filter_spec.rb
#
# run using:
# bundle exec rspec simple_filter_spec.rb
require "logstash/devutils/rspec/spec_helper"
LogStash::Environment::LOGSTASH_HOME = `gem which logstash-core`
module LogStash::Environment
unless self.method_defined?(:pattern_path)
def pattern_path(path)
::File.join(LOGSTASH_HOME, "patterns", path)
end
end
end
require "logstash/filters/grok"
describe LogStash::Filters::Grok do
config <<-CONFIG
filter {
grok {
match => { "message" => "%{SYSLOGLINE}" }
singles => true
overwrite => [ "message" ]
}
}
CONFIG
sample "Mar 16 00:01:25 evita postfix/smtpd[1713]: connect from camomile.cloud9.net[168.100.1.3]" do
insist { subject["tags"] }.nil?
insist { subject["logsource"] } == "evita"
insist { subject["timestamp"] } == "Mar 16 00:01:25"
insist { subject["message"] } == "connect from camomile.cloud9.net[168.100.1.3]"
insist { subject["program"] } == "postfix/smtpd"
insist { subject["pid"] } == "1713"
end
end
而且我的Gemfile看起来像这样:
source 'https://www.rubygems.org'
platform :jruby do
gem 'pry'
gem 'rspec'
gem 'logstash-core'
gem 'logstash-devutils'
gem 'logstash-filter-grok'
end
可以使用Logstash-Tester - 一个小工具,单元测试logstash过滤器和模式。您使用json编写测试用例,logstash-tester使用docker容器在logstash上运行它们。 (免责声明:我写的工具)
$ git clone https://github.com/elastic/logstash
$ cd logstash
$ git checkout 2.1
$ rake bootstrap
$ rake test:install-core
而不是检查出来的2.1分支,你应该检查出logstash的标记的版本,你会实际运行,例如v2.3.2(注意前面的“v”)。
运行上述命令后,可以在logstash回购中运行bin/rspec /some/path/your_filter_spec.rb
。
重要:我发现编码行# encoding: utf-8
是必需的,否则匹配将失败。
样品测试文件:由gmile给出
# encoding: utf-8
require "spec_helper"
describe "simple request log" do
config (<<-CONFIG)
filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
CONFIG
sample '55.3.244.1 GET /index.html 15824 0.043' do
insist { subject['client'] } == '55.3.244.1'
insist { subject['method'] } == 'GET'
insist { subject['request'] } == '/index.html'
insist { subject['bytes'] } == '15824'
insist { subject['duration'] } == '0.043'
end
end
你能在这里的文件路径详细点吗?您是在Ansible控制器上还是在Logstash服务器上运行测试?即使用你的'Gemfile'例子,我无法运行你的示例spec文件。 – conorsch
我有这个错误'NoMethodError:未定义的方法'pattern_path'为LogStash :: Environment:Module'。似乎必须将其添加到文件:https:// github。com/logstash-plugins/logstash-filter-grok/pull/73/commits/0be653da88887b2b2413de543d03372a32244905 – Keymon
@conorsch查看我的回答 – Kelvin