面对java.lang.OutOfMemoryError：使用Camel Aggregator时超出GC开销限制

Question

我正面临java.lang.OutOfMemoryError：使用聚合生成的文件超出了GC开销限制。下面是我的路线

<route id="sendToFileRoute"> 
     <from uri="direct:sendToFile" /> 
     <choice> 
      <when> 
       <!-- check to ignore if from LIVEFEED --> 
       <simple>${property.FileName} != 'LIVEFEED'</simple> 
       <to uri="bean:intermediateFileProcessor?method=processSNE" /> 
       <camel:aggregate completionSize="300" strategyRef="aggrMessToFile" forceCompletionOnStop="true" completionTimeout="5000" > 
        <camel:correlationExpression> 
         <camel:simple>${property.stageFileName}</camel:simple> 
        </camel:correlationExpression> 
       <camel:to uri="file://{{intermediate.dir.path}}?fileExist=Append&amp;fileName=$simple{property.stageFileName}" /> 
       </camel:aggregate> 
      </when> 
     </choice> 
    </route> 

生成的文件是一样的东西 - V2_10.246.84.69_INI_TIMER_US_20150528_102000514_32.xml

下面是错误日志

2015:05:28:10:43:09.552 [Camel (locationContext) thread #2 - file:///cert/pubsub/location/mount/] ERROR com.walmart.eim.location.common.handle 
rs.ExceptionHandler RequestId - 2845692472690849 : FileName - V2_10.246.84.69_INI_TIMER_US_20150528_102000514_32.xml - 
2015:05:28:10:43:37.107 [Camel (locationContext) thread #7 - timer://deltaTimer] ERROR com.walmart.eim.location.common.handlers.ExceptionHandl 
er - 
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: [Body is null]] 
     at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1360) 
     at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:272) 
     at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:88) 
     at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) 
     at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:139) 
     at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:64) 
     at java.util.TimerThread.mainLoop(Timer.java:555) 
     at java.util.TimerThread.run(Timer.java:505) 
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded 
2015:05:28:10:43:43.307 [Camel (locationContext) thread #2 - file:///cert/pubsub/location/mount/] ERROR com.walmart.eim.location.common.handle 
rs.ExceptionHandler RequestId - 2845692472690849 : FileName - V2_10.246.84.69_INI_TIMER_US_20150528_102000514_32.xml - 
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[V3ConfidES_10.246.84.69_INITIAL_TIMER_ 
US_20150528_102000514_32.xml] 
     at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1360) 
     at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:272) 
     at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:224) 
     at org.apache.camel.processor.Splitter.process(Splitter.java:98) 
     at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) 
     at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:352) 
     at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:199) 
     at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:165) 
     at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) 
     at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114) 
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) 
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
     at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded 
2015:05:28:10:43:43.353 [Camel (locationContext) thread #7 - timer://deltaTimer] ERROR com.walmart.eim.location.common.handlers.ExceptionHandl 
er RequestId - 2846346936275989 - 
java.lang.NullPointerException 
     at org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:148) 
     at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280) 
     at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369) 
     at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342) 
     at org.apache.velocity.runtime.RuntimeInstance.render(RuntimeInstance.java:1378) 
     at org.apache.velocity.runtime.RuntimeInstance.evaluate(RuntimeInstance.java:1314) 
     at org.apache.velocity.app.VelocityEngine.evaluate(VelocityEngine.java:272) 
     at org.apache.camel.component.velocity.VelocityEndpoint.onExchange(VelocityEndpoint.java:191) 
     at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71) 
     at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) 
     at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110) 
     at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) 
     at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) 
     at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:103) 
     at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) 
     at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:56) 
     at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110) 
     at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) 
     at org.apache.camel.processor.FatalFallbackErrorHandler.process(FatalFallbackErrorHandler.java:42) 
     at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:839) 
     at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:337) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) 
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) 
     at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) 
     at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:139) 
     at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:64) 
     at java.util.TimerThread.mainLoop(Timer.java:555) 
     at java.util.TimerThread.run(Timer.java:505) 

能有人帮我上解决这个？

Answer 1

Camel Aggregrator2正在运行内存，可能是由于大型数据集。 在这种情况下，最好使用基于数据库的聚合策略，使用 - org.apache.camel.processor.aggregate.jdbc.JdbcAggregationRepository

这将是一个更具可扩展性的选项。 这里有一篇很棒的文章：http://labs.bsb.com/2011/04/jdbc-persistence-for-camel-aggregator/