2017-02-15 65 views
0

我试图在运行ubuntu 16.04的aws ec2实例上使用systemd运行logstash。我在机器上安装了heroku工具栏。正常运行管道(通过bin/logstash.bat)可以正常工作,并且事件被摄入(然而在几分钟后,得到“请求超时”错误并且管道停止,这是一个单独的问题)。在Ubuntu(logstash 5.2.1)上运行的logstash-input-heroku服务

但是,当我尝试在systemd上运行服务时,我得到错误,不知道这两种类型的错误是否有关系。第一个是SSL错误:

Error: no cipher match (OpenSSL::SSL::SSLError)

[2017-02-15T13:08:44,037][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin. Plugin: "xxxxxx", codec=>"^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})", what=>"previous", id=>"032c3b317ae49982945ec7e8fbf11224be98f237-3", enable_metric=>true, negate=>false, charset=>"UTF-8", multiline_tag=>"multiline", max_lines=>500, max_bytes=>10485760>, id=>"032c3b317ae49982945ec7e8fbf11224be98f237-4", enable_metric=>true>

第二个问题是Heroku的工具区似乎被提示输入凭据:

Feb 15 13:08:43 ip-10-0-1-216 logstash[4402]: Enter your Heroku credentials.

Feb 15 13:08:43 ip-10-0-1-216 logstash[4402]: Email: Password (typing will be hidden):

我logstash配置:

input { 
    heroku { 
     app => "xxx-1" 
     codec => multiline { 
      pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})" 
      what => "previous" 
     } 
    } 
    heroku { 
     app => "xxx-2" 
     codec => multiline { 
      pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})" 
      what => "previous" 
     } 
    } 
    heroku { 
     app => "xxx-3" 
     codec => multiline { 
      pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})" 
      what => "previous" 
     } 
    } 
    heroku { 
     app => "xxx-4" 
     codec => multiline { 
      pattern => "^%{TIMESTAMP_ISO8601} %{WORD}\[\w+(\.\d+)?\]:(\s{3,}| \})" 
      what => "previous" 
     } 
    } 
} 

filter { 
    grok { 
    break_on_match => true 
    patterns_dir => ["./grok_patterns"] 
    match => { "message" => [ 
     "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:heroku_source}\[%{DYNO:dyno}\]: %{LEVEL:level}: HTTP %{OPT_NOT_SPACE_COMMA:organization}, %{OPT_NOT_COMMA:user}, %{OPT_NOT_COMMA:device}, %{WORD:method} %{ENDPOINT:endpoint}%{QUERY:query} \[%{INT:responseCode:int}\].*? \(p%{INT:nodeProcess:int}\) \(%{INT:responseTime:int}ms\).*$", 
     "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:heroku}\[%{WORD:component}\]: at=\w+ method=%{WORD:method} path=\"%{ENDPOINT:endpoint}\??%{QUERY:query}\" .*?fwd=\"%{IP:site_ip}\" dyno=%{DYNO:dyno} .*?service=%{INT:responseTime:int}ms status=%{INT:responseCode:int} bytes=%{INT:sizeBytes:int}.*?$", 
     "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:heroku_source}\[%{DYNO:dyno}\]: (?<data>.*)" 
    ] } 
    add_field => { "endpoint_template" => "%{endpoint}" } 
    } 
    mutate { 
    gsub => ["endpoint_template", "[0-9a-f]{24}", "ID"] 
    add_field => { "type" => "heroku" } 
    } 
    if ![heroku_source] { 
    geoip { 
     source => "site_ip" 
    } 
    mutate { 
     add_field => { "heroku_source" => "heroku" } 
    } 
    } 
} 

output { 
    elasticsearch { 
     hosts => [ "aws-es-endpoint:443" ] 
     ssl => true   
    } 
} 

(我肯定可以改进)

我试过以root身份运行服务,但结果是一样的。只是为了澄清,这个工作:

/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ 

虽然这并不:

sudo systemctl start logstash 

这是一个干净的安装logstash 5.2.1以下的procedures on elastic的。 Systemd也根据their procedures运行,以便它执行与我手动执行相同的命令。 cat logstash.service输出:

[Unit] 
Description=logstash 

[Service] 
Type=simple 
User=logstash 
Group=logstash 
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. 
# Prefixing the path with '-' makes it try to load, but if the file doesn't 
# exist, it continues onward. 
EnvironmentFile=-/etc/default/logstash 
EnvironmentFile=-/etc/sysconfig/logstash 
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" 
Restart=always 
WorkingDirectory=/ 
Nice=19 
LimitNOFILE=16384 

[Install] 
WantedBy=multi-user.target 

(结果是一样的,当我注释掉用户及以上组)

回答

0

只是为了记录在案,原来的问题是,systemd没有看到Heroku的凭据。我问了一个similar question on AskUbuntu。解决方案是将ubuntu主目录添加到logstash服务,以便它可以访问凭据。通过编辑/etc/systemd/system中的logstash.service文件并在那里设置Environment="Home=/home/ubuntu"来完成。最终logstash.service文件的

实施例:

[Unit] 
Description=logstash 

[Service] 
Type=simple 
User=logstash 
Group=root 
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. 
# Prefixing the path with '-' makes it try to load, but if the file doesn't 
# exist, it continues onward. 
Environment="HOME=/home/ubuntu" 
EnvironmentFile=-/etc/default/logstash 
EnvironmentFile=-/etc/sysconfig/logstash 
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" 
Restart=always 
WorkingDirectory=/ 
Nice=19 
LimitNOFILE=16384 

[Install] 
WantedBy=multi-user.target 
相关问题