ClassLoader Leaks by Oracle

I recently had trouble with a web application deployed on Tomcat that leaked its ClassLoader every time it was redeployed resulting in OutOfMemoryErrors after a few redeployments. This is quite nasty if you plan to do continuous deployment and don’t want to restart the servlet container with each deployment.

Recent versions of Tomcat include some code that makes you aware of problems when you undeploy the application:

SEVERE: The web application [] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped.
 To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
SEVERE: The web application [] appears to have started a thread named [Thread-14] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-15] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-16] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-17] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-18] but has failed to stop it. This is very likely to create a memory leak.

As you can see Tomcat managed to unregister the JDBC driver that the application had failed to unregister but could do nothing regarding the threads that had been started but not stopped.

I ran the application with YourKit attached to check that the WebappClassLoader had actually leaked and to see what those threads were that prevented it from being garbage collected. The “Paths from GC Roots” view in YourKit is well suited for this:

ONS Leaking Threads

As you can see there are four threads from ONS that prevent the ClassLoader from being garbage collected: oracle.ons.SenderThreads and oracle.ons.ReceiverThreads.

I wrote a small ServletContextListener that shuts down ONS to get rid of them. After that I noticed that Oracle registered a OracleDiagnosabilityMBean that I had to unregister. Finally I made sure the JDBC drivers that the application had registered were properly unregistered from DriverManager.

With those changes in place the application undeployed well and was fully garbage collected.

Here is the code:

import oracle.ons.ONS;
import oracle.ons.SenderThread;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.ReflectionUtils;

import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import java.lang.management.ManagementFactory;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.sql.Driver;
import java.sql.DriverManager;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.Hashtable;
import java.util.List;

public class CleanUpListener implements ServletContextListener {
    private Logger logger = LoggerFactory.getLogger(getClass());

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        // do nothing
    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        shutdownOns();
        deregisterJdbcDrivers();
    }

    @SuppressWarnings("unchecked")
    private void shutdownOns() {
        logger.info("Shutting down ONS");
        final Method getRunningONS = ReflectionUtils.findMethod(ONS.class, "getRunningONS");
        final Method shutdown = ReflectionUtils.findMethod(ONS.class, "shutdown");
        ReflectionUtils.makeAccessible(getRunningONS);
        ReflectionUtils.makeAccessible(shutdown);
        final ONS ons = (ONS) ReflectionUtils.invokeMethod(getRunningONS, null);
        if (ons == null) {
            return;
        }
        
        ReflectionUtils.invokeMethod(shutdown, ons);

        final Field senders = ReflectionUtils.findField(ONS.class, "senders");
        ReflectionUtils.makeAccessible(senders);
        final List<SenderThread> senderThreads = (List<SenderThread>) ReflectionUtils.getField(senders, ons);
        if (senderThreads == null) {
            return;
        }
        
        final Method stopThread = ReflectionUtils.findMethod(SenderThread.class, "stopThread");
        ReflectionUtils.makeAccessible(stopThread);
        for (SenderThread senderThread : senderThreads) {
            ReflectionUtils.invokeMethod(stopThread, senderThread);
        }
    }
    
    private void deregisterJdbcDrivers() {
        logger.info("Deregistering JDBC Drivers");
        final Enumeration<Driver> driverEnumeration = DriverManager.getDrivers();
        final List<Driver> drivers = new ArrayList<Driver>();
        while (driverEnumeration.hasMoreElements()) {
            drivers.add(driverEnumeration.nextElement());
        }

        for (Driver driver : drivers) {
            if (driver.getClass().getClassLoader() != getClass().getClassLoader()) {
                logger.debug("Not deregistering {} as it does not originate from this webapp", driver.getClass().getName());
                continue;
            }
            try {
                DriverManager.deregisterDriver(driver);
                logger.debug("Deregistered JDBC driver '{}'", driver.getClass().getName());
                if ("oracle.jdbc.OracleDriver".equals(driver.getClass().getName())) {
                    deregisterOracleDiagnosabilityMBean();
                }
            } catch (Throwable e) {
                logger.error("Deregistration error", e);
            }
        }
    }

    private void deregisterOracleDiagnosabilityMBean() {
        final ClassLoader cl = Thread.currentThread().getContextClassLoader();
        try {
            final MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
            final Hashtable<String, String> keys = new Hashtable<String, String>();
            keys.put("type", "diagnosability");
            keys.put("name", cl.getClass().getName() + "@" + Integer.toHexString(cl.hashCode()).toLowerCase());
            mbs.unregisterMBean(new ObjectName("com.oracle.jdbc", keys));
            logger.info("Deregistered OracleDiagnosabilityMBean");
        } catch (javax.management.InstanceNotFoundException e) {
            logger.debug("Oracle OracleDiagnosabilityMBean not found", e);
        } catch (Throwable e) {
            logger.error("Oracle JMX unregistration error", e);
        }
    }
}

Will the BEA Acquisition Push Spring Framework?


BEA is well known for its J2EE application server Weblogic and recently extended its product line to a SOA stack branded Liquid. Oracle has its own J2EE compontents mainly derived from Orion Server and branded as OC4J. They also have a SOA stack called Fusion.

So you might ask what the future of these products will be.

Oracle already acquired three different business application suites in the past: J. D. Edwards, PeopleSoft and Siebel. Does it make sense to develop three different product lines of business application suites and at least two different product lines of Java EE and SOA middleware? Probably not.

This uncertain future may make companies reevaluate their current technology stack for middleware applications. They will notice that there is an alternative beyond the IBM Websphere dinosaur and JBoss which is now RedHat. The alternative is Apache Tomcat and Spring Framework mixed with some Terracotta if you need distributed shared data.

Spring basically allows you to assemble your own application server. Transaction management, security, remoting, O/R mapping, clustering – just add what you need when you need it.
Basically there is not much left where Spring does not offer a proven solution. On the other hand the features where commercial J2EE servers have been strong in the past (compared to other stacks) are becoming less important:

  • EJB remoting will be replaced by web services
  • entity bean style O/R mapping has already been replaced by Hibernate (where EJB3 tries to catch on)
  • load distribution and fail-over is easy in a world of HTTP or JMS based web services
  • clustering of data is less important in the stateless world of services

In the end the uncertainty caused by Oracle’s acquisition of BEA could very well further increase the growth of Spring and the open source components it makes so easy to combine.