Glassfish 4灰熊线程沉重的CPU使用率

Question

我有一个在Glassfish 4（4.1版本13），JDK 1.7 update 67和AWS Linux AMI上运行的泽西岛应用程序，我注意到在运行它几个小时后，CPU使用率上升，即使客户停下来也会保持住。

运行“top -H”标识具有较高CPU使用率的2个http-listener-1内核线程（总数为16）。然后，我花了线程转储来检查这两个线程：

"http-listener-1-kernel(3) SelectorRunner" daemon prio=10 tid=0x00007fbc68251000 nid=0xaee runnable [0x00007fbcb55ce000] 
    java.lang.Thread.State: RUNNABLE 
     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) 
     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) 
     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) 
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) 
     - locked <0x000000060263ad88> (a sun.nio.ch.Util$2) 
     - locked <0x000000060263ad78> (a java.util.Collections$UnmodifiableSet) 
     - locked <0x00000006025fb068> (a sun.nio.ch.EPollSelectorImpl) 
     at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106) 
     at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:114) 
     at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338) 
     at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) 
     at java.lang.Thread.run(Thread.java:745) 


"http-listener-1-kernel(8) SelectorRunner" daemon prio=10 tid=0x00007fbc6825b800 nid=0xaf3 runnable [0x00007fbcb50c9000] 
    java.lang.Thread.State: RUNNABLE 
     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) 
     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) 
     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) 
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) 
     - locked <0x00000006026648b8> (a sun.nio.ch.Util$2) 
     - locked <0x00000006026648a8> (a java.util.Collections$UnmodifiableSet) 
     - locked <0x0000000602664790> (a sun.nio.ch.EPollSelectorImpl) 
     at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106) 
     at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:114) 
     at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338) 
     at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) 
     at java.lang.Thread.run(Thread.java:745) 

有趣的是，不消耗CPU的其他线程使用的sun.nio.ch.SelectorImpl不同的方法（选择而非selectNow）：

"http-listener-1-kernel(7) SelectorRunner" daemon prio=10 tid=0x00007fbc68259800 nid=0xaf2 runnable [0x00007fbcb51ca000] 
    java.lang.Thread.State: RUNNABLE 
     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) 
     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) 
     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) 
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) 
     - locked <0x0000000602665f20> (a sun.nio.ch.Util$2) 
     - locked <0x0000000602665f10> (a java.util.Collections$UnmodifiableSet) 
     - locked <0x0000000602665df8> (a sun.nio.ch.EPollSelectorImpl) 
     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) 
     at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:112) 
     at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338) 
     at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) 
     at java.lang.Thread.run(Thread.java:745) 

然后我再次打开客户端，让他们再运行几个小时，然后再次停止。果然，“top -H”表明CPU使用率再次增加。另一个线程转储揭示了一个更加线程切换到“selectNow”，现在（与其他2一起）采取更多的CPU时间：

"http-listener-1-kernel(6) SelectorRunner" daemon prio=10 tid=0x00007fbc68257800 nid=0xaf1 runnable [0x00007fbcb52cb000] 
    java.lang.Thread.State: RUNNABLE 
     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) 
     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) 
     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) 
     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) 
     - locked <0x00000006026675c0> (a sun.nio.ch.Util$2) 
     - locked <0x00000006026675b0> (a java.util.Collections$UnmodifiableSet) 
     - locked <0x0000000602667498> (a sun.nio.ch.EPollSelectorImpl) 
     at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106) 
     at org.glassfish.grizzly.nio.DefaultSelectorHandler.select(DefaultSelectorHandler.java:114) 
     at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:338) 
     at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) 
     at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) 
     at java.lang.Thread.run(Thread.java:745) 

看来，东西导致线程调用的“selectNow”方法sun.nio.ch.SelectorImpl，一旦它们进入那里，CPU使用率将大量增加，并且不会减少，除非服务器重新启动。

这是一个已知的问题？这可能是由我的代码造成的？

感谢您的帮助！

Answer 1

可能是这个问题：

https://java.net/jira/browse/GLASSFISH-21211

没有为灰熊包含在这一问题的补丁，但我没有测试它。