2010-11-14 178 views
0

我有我的网站停机时间问题。我在Linux Redhat上安装了apache 2 + mod_jk + tomcat 5.5安装程序。最近刚刚开始让我的网站每天停机一次。只有重新启动tomcat后,网站才会恢复正常。在Tomcat网站下降 - 在mod_jk(apache + tomcat)日志中的错误

应用程序错误日志(对不起,隐私信息隐藏在日志):

 
2010-11-14 00:40:13,726 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet action threw exception 
java.lang.OutOfMemoryError: unable to create new native thread 
    at java.lang.Thread.start0(Native Method) 
    at java.lang.Thread.start(Thread.java:574) 
    at ******.exchange.adapter.dispatcher.impl.XMLCommandDispatcher.sendCommand(XMLCommandDispatcher.java:109) 
    at ******.exchange.adapter.star******.api.impl.******ApiImpl.******PricingAvailability(******ApiImpl.java:356) 
    at ******.exchange.adapter.star******.impl.Concrete******GatewayImpl.find******Promotion(Concrete******GatewayImpl.java:123) 
    at ******.exchange.service.impl.Concrete******ManagerImpl.find******Promotion(Concrete******ManagerImpl.java:199) 
    at sun.reflect.GeneratedMethodAccessor1142.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:585) 
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:282) 
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:155) 
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:122) 
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56) 
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144) 
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:174) 
    at $Proxy16.find******Promotion(Unknown Source) 
    at ******.ibe.servlet.crdms.search.******SearchPromotionServlet******.handleRequest(******SearchPromotionServlet******.java:75) 
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44) 
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:595) 
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:543) 
    at org.springframework.web.servlet.FrameworkServlet.serviceWrapper(FrameworkServlet.java:390) 
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:341) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) 
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174) 
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) 
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) 
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) 
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) 
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) 
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) 
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) 
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) 
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) 
    at java.lang.Thread.run(Thread.java:595) 
2010-11-14 00:40:43,762 ERROR org.apache.catalina.core.ApplicationDispatcher [invoke] - Servlet.service() for servlet jsp threw exception 
2010-11-14 00:40:43,805 ERROR org.apache.catalina.core.StandardWrapperValve [invoke] - Servlet.service() for servlet jsp threw exception 
java.lang.OutOfMemoryError: unable to create new native thread 
    at java.lang.Thread.start0(Native Method) 
    at java.lang.Thread.start(Thread.java:574) 
    at java.util.Timer.(Timer.java:154) 
    at java.util.Timer.(Timer.java:122) 
    at com.mysql.jdbc.Connection.(Connection.java:1441) 
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) 
    at java.sql.DriverManager.getConnection(DriverManager.java:525) 
    at java.sql.DriverManager.getConnection(DriverManager.java:171) 
    at org.apache.jsp.frameset_005f******_005fbottom_005f1_jsp._jspService(frameset_005f******_005fbottom_005f1_jsp.java:88) 
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) 
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331) 
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329) 
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) 
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174) 
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) 
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) 
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) 
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) 
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) 
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) 
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) 
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) 
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) 
    at java.lang.Thread.run(Thread.java:595) 
2010-11-14 01:54:25,819 ERROR org.apache.catalina.core.ApplicationDispatcher [invoke] - Servlet.service() for servlet jsp threw exception 

mod_jk的时刻当Tomcat下来的日志条目:

 
[Sun Nov 14 00:59:52 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 
[Sun Nov 14 00:59:52 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) 
[Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 
[Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) 
[Sun Nov 14 01:00:01 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed 
[Sun Nov 14 01:00:01 2010] worker1 mydomain.com 271.622400 

[Sun Nov 14 01:00:01 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 
[Sun Nov 14 01:00:01 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) 
[Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 
[Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) 
[Sun Nov 14 01:00:06 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed 
[Sun Nov 14 01:00:06 2010] worker1 mydomain.com 86.999946 

[Sun Nov 14 01:00:06 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 
[Sun Nov 14 01:00:06 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) 
[Sun Nov 14 01:00:10 2010] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the response message from tomcat, network problems or tomcat is down (127.0.0.1:8009), err=-104 
[Sun Nov 14 01:00:10 2010] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet) 
[Sun Nov 14 01:00:10 2010] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=worker1 failed 
[Sun Nov 14 01:00:10 2010] worker1 mydomain.com 39.000187 

Meesages in/var/log/messages:

 
Nov 14 00:59:40 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:00:47 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:01:49 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:02:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:04:03 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:05:06 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:06:13 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:07:14 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:08:15 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:09:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:10:23 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:11:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:12:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:13:29 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:14:56 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:16:04 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:17:19 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:18:22 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:19:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:20:26 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:21:36 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:22:37 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:23:48 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:24:50 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:25:54 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:26:57 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:29:08 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 01:30:10 nemo kernel: possible SYN flooding on port 8009. Sending cookies. 
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/mysql.sock (var_lib_t). For complete SELinux messages. run sea$ 
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "getattr" to /var/lib/mysql/nemo.me-tech.com.my.err (var_lib_t). For complete SELinux mess$ 
Nov 14 03:00:01 nemo setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./time_zone_name.frm (var_lib_t). For complete SELinux messages. run sealert -l $ 

阿帕奇CONF:

 


# prefork MPM 
# StartServers: number of server processes to start 
# MinSpareServers: minimum number of server processes which are kept spare 
# MaxSpareServers: maximum number of server processes which are kept spare 
# MaxClients: maximum number of server processes allowed to start 
# MaxRequestsPerChild: maximum number of requests a server process serves 
‹IfModule prefork.c› 
StartServers   5 
MinSpareServers  5 
MaxSpareServers  10 
MaxClients   256 
MaxRequestsPerChild 0 
‹/IfModule› 

# worker MPM 
# StartServers: initial number of server processes to start 
# MaxClients: maximum number of simultaneous client connections 
# MinSpareThreads: minimum number of worker threads which are kept spare 
# MaxSpareThreads: maximum number of worker threads which are kept spare 
# ThreadsPerChild: constant number of worker threads in each server process 
# MaxRequestsPerChild: maximum number of requests a server process serves 
‹IfModule worker.c› 
StartServers   2 
MaxClients   256 
MinSpareThreads  25 
MaxSpareThreads  75 
ThreadsPerChild  25 
MaxRequestsPerChild 0 
‹/IfModule› 

的Apache mod_jk的worker.properties:

 
worker.list=worker1 
worker.worker1.type=ajp13 
worker.worker1.host=127.0.0.1 
worker.worker1.port=8009 
worker.worker1.lbfactor=1 
worker.worker1.socket_keepalive=1 
worker.worker1.recycle_timeout=180 
worker.worker1.sticky_session=False 

的Tomcat的server.xml(AJP连接器的mod_jk):

 

    ‹Connector className="org.apache.coyote.tomcat5.CoyoteConnector" 
       port="8009" 
       protocol="AJP/1.3" 
       enableLookups="true" 
       redirectPort="8443" 
       acceptCount="10" 
       debug="0" 
       connectionTimeout="5000" 
       useURIValidationHack="false" 
       maxPostSize="10000000" 
       maxThreads="256" 
       maxSpareThreads="10" 
       URIEncoding="UTF-8" 
       protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/› 

任何帮助或建议将不胜感激。

非常感谢

回答

1

根据this你应该尝试设置您的JAVA_OPTS环境变量,以减少被Tomcat用来给OS一些喘息空间堆栈和/或堆的大小。