java.lang.OutOfMemoryError: PermGen space

For asynchronous messaging we are using Active MQ in a few web applications deployed to Apache Tomcat.

This works quite well until you start redeploying the applications multiple times without restarting Tomcat. After a few redeployments you will inevitably hit a java.lang.OutOfMemoryError: PermGen space. This usually indicates a memory leak caused by the class loader of the web application not being garbage collected.

I've used Lambda Probe to get an overview of what's going on: Indeed PermGen usage increased with each redeployment until it finally hit the wall after 4 minutes (redeploying every 30 seconds) as you can see on the first chart. Having a look at the memory with YourKit Java Profiler also showed an additional WebappClassLoader after each redeployment. The problem with the leaked WebappClassLoader is that it contains hard references to all classes that it has loaded – for our webapp more than 2000 – that pollute the PermGen.

What is causing the WebappClassLoader to leak?

There are two common causes:

• Threads started by the webapp that are not properly terminated
• References to your objects or classes from outside of your webapp

As it turned out ActiveMQ uses a few internal "ActiveMQ Scheduler" threads that are part of a ThreadGroup of a maximum of 5 threads. These threads are not properly cleaned when ActiveMQ is shut down. AMQ-1214 describes the issue.

What can we do to prevent the leak?

The ThreadGroup must be shut down. Fortunately the clockDaemon property is publicly accessible so the easiest solution was to add a clean up bean to my spring context that calls it shutdown() method when the context is destroyed.

import org.apache.activemq.thread.Scheduler;
import org.springframework.beans.factory.DisposableBean;

public class ActiveMQCleaner implements DisposableBean
{
    public void destroy() throws Exception
    {
        Scheduler.clockDaemon.shutdown();
    }
}

As you can see in the second chart this fixed the problem. More than 20 minutes and 50 redeployment cylces later still no OutOfMemoryError.

If you are interested in other causes of PermGen leaks and strategies to hunt them have a look at Frank Kieviet's Blog and his JavaOne Presentation.