SpringBoot - > Logback - > Logstash

Question

事件不会从我的Spring Boot应用程序发送到logstash。这里是我的logback.xml文件：

<configuration> 
    <appender name="STASH-C" class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender"> 
     <destination>arc-poc01:5044</destination> 
     <encoder class="net.logstash.logback.encoder.LogstashAccessEncoder" /> 
     <keepAliveDuration>5 minutes</keepAliveDuration> 
    </appender> 
    <appender name="STASH-B" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> 
    <destination>arc-poc01:5045</destination> 
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder" > 
     <providers> 
      <timestamp/> 
      <version/> 
      <loggerName/> 
      <pattern> 
       <pattern> 
        { 
        "custom_constant": "cfg", 
        "level": "%level", 
        "thread": "%thread", 
        "message": "%message" 
        } 
       </pattern> 
      </pattern> 
     </providers> 
    </encoder> 
    <keepAliveDuration>5 minutes</keepAliveDuration> 
    </appender> 
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> 
    <!-- encoders are assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder 
     by default --> 
    <encoder> 
     <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 
     </pattern> 
    </encoder> 
    </appender> 
    <logger name="com.gw.test" level="INFO" /> 
    <logger name="org.springframework" level="INFO" /> 
    <logger name="com.netflix.astyanax" level="INFO" /> 

    <root level="DEBUG"> 
    <appender-ref ref="STASH-B" /> 
    <appender-ref ref="STASH-C" /> 
    <appender-ref ref="STDOUT" /> 
    </root> 
</configuration> 

这里是我的logstash管道配置：

input { 
    tcp { 
     port => 5044 
     codec => json 
     data_timeout => -1 
    } 
    log4j { 
     mode => "server" 
     host => "0.0.0.0" 
     port => 5045 
     type => "log4j" 
     codec => json 
    } 
} 

output { 
    stdout { codec => rubydebug } 
    elasticsearch { 
     hosts => [ "http://arc-poc01:9200" ] 
    } 
} 

当我运行正在生成的应用程序的控制台输出作为预期，但没有事件被发送到logstash。我跑卷曲-XGET的“localhost：9600/_node /统计/管道漂亮”，以验证流量，这里是输出：

{ 
    "host" : "55357b6f0969", 
    "version" : "5.4.1", 
    "http_address" : "0.0.0.0:9600", 
    "id" : "65153f2f-10af-48c3-9be5-5db0913bf7d8", 
    "name" : "55357b6f0969", 
    "pipeline" : { 
    "events" : { 
     "duration_in_millis" : 0, 
     "in" : 0, 
     "filtered" : 0, 
     "out" : 0, 
     "queue_push_duration_in_millis" : 0 
    }, 
    "plugins" : { 
     "inputs" : [ { 
     "id" : "45e636052face5ff7a0b8cb463fa2b88c59c5697-2", 
     "events" : { 
      "out" : 0, 
      "queue_push_duration_in_millis" : 0 
     }, 
     "name" : "log4j" 
     }, { 
     "id" : "45e636052face5ff7a0b8cb463fa2b88c59c5697-1", 
     "events" : { 
      "out" : 0, 
      "queue_push_duration_in_millis" : 0 
     }, 
     "name" : "tcp" 
     } ], 
     "filters" : [ ], 
     "outputs" : [ { 
     "id" : "45e636052face5ff7a0b8cb463fa2b88c59c5697-3", 
     "name" : "stdout" 
     }, { 
     "id" : "45e636052face5ff7a0b8cb463fa2b88c59c5697-4", 
     "name" : "elasticsearch" 
     } ] 
    }, 
    "reloads" : { 
     "last_error" : null, 
     "successes" : 0, 
     "last_success_timestamp" : null, 
     "last_failure_timestamp" : null, 
     "failures" : 0 
    }, 
    "queue" : { 
     "type" : "memory" 
    }, 
    "id" : "main" 
    } 
} 

有两者均显示我的春天启动应用程序的日志了任何错误或Logstash。在Spring Boot应用程序中配置logback时，是否存在特定的问题？我花了2天时间研究这个问题，并用尽了想法。任何提示如何我可以解决这个问题将不胜感激！

Answer 1

我还应该添加到我的原始描述中，我在通过Marathon脚本部署在Mesos上的Docker容器中运行ELK堆栈。下面是最终帮我把我的应用程序通过ELK管道流的日志：

1）改变logstash管道的定义如下：

input { 
    tcp { 
     port => 5045 
     codec => json 
    } 
} 
output { 
    stdout { codec => rubydebug } 
    elasticsearch { 
     hosts => [ "http://arc-poc01:9200" ] 
    } 
} 

2）logback.xml文件作为配置Logstash附加器如下：

<appender name="STASH-B" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> 
    <destination>arc-poc01:5045</destination> 
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder" > 
     <providers> 
      <timestamp/> 
      <version/> 
      <loggerName/> 
      <pattern> 
       <pattern> 
        { 
        "custom_constant": "cfg", 
        "level": "%level", 
        "thread": "%thread", 
        "message": "%message" 
        } 
       </pattern> 
      </pattern> 
     </providers> 
    </encoder> 
    <keepAliveDuration>5 minutes</keepAliveDuration> 
    </appender> 

3）获取从Logstash附加器相关的调试跟踪中流露出Logstash连接我的客户端应用程序上上下下通过包括StatusListener在logback.xml文件，如下所示：

<statusListener class="ch.qos.logback.core.status.OnConsoleStatusListener" /> 

日志条目指示正在建立logstash连接，并立即由logstash服务器关闭。我认为问题是由管道中定义的编解码器配置与客户端应用程序端的logstash appender发送的内容类型之间明显不匹配造成的。

Answer 2

尝试从输入移动JSON编解码器进行过滤： -

input { 
    tcp { 
     port => 5045 
    } 
} 
filter 
{ 
     json { 
      source => "message" 
      remove_field => "message" 
      } 
} 
output { 
    stdout { codec => rubydebug } 
    elasticsearch { 
     hosts => [ "http://arc-poc01:9200" ] 
    } 
}