C3P0 apparent deadlock when the threads are all empty?

java connection-pooling c3p0
Steven Schlansker picture Steven Schlansker · Sep 16, 2010 · Viewed 65.3k times · Source

I'm using C3P0 as a connection pool in Tomcat, and I'm seeing very worrying errors:

2010-09-16 13:25:00,160 [Timer-0] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner  - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@43502400 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2010-09-16 13:25:01,407 [Timer-0] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner  - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@43502400 -- APPARENT DEADLOCK!!! Complete Status:
  Managed Threads: 10
  Active Threads: 0
  Active Tasks:
  Pending Tasks:
    com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@6e4151a7
  Pool thread stack traces:
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,main]
    java.lang.Object.wait(Native Method)
    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
  Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,main]
    java.lang.Object.wait(Native Method)

... many more, exact same stack trace

Line 534 is:

 while (true) {
   Runnable myTask;
   synchronized ( ThreadPoolAsynchronousRunner.this ) {
     while ( !should_stop && pendingTasks.size() == 0 )
       ThreadPoolAsynchronousRunner.this.wait( POLL_FOR_STOP_INTERVAL ); // <- here
     if (should_stop) ...

It looks very much like all the threads are idle. They're waiting for work. 0 active threads, and only 1 task to complete. Any clues on what's going wrong?

Here's the configuration:

ds.setUser(userName);
ds.setPassword(password);
ds.setMaxPoolSize(16);
ds.setMaxConnectionAge(1800);
ds.setAcquireRetryAttempts(4);
ds.setMaxIdleTime(900);
ds.setNumHelperThreads(10);
ds.setCheckoutTimeout(1000);

Answer

eebbesen picture eebbesen · Aug 12, 2013

I just experienced a similar issue against an Oracle database, but in my case Managed Thread and Active Thread counts were the same.

    Managed Threads: 3
    Active Threads: 3

For me it was actually an authentication error but appeared as the APPARENT DEADLOCK error because of the way I was doing login auditing. 

    2013-08-12 11:29:04,910 [Timer-4] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@34996454 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    2013-08-12 11:29:04,914 [Timer-4] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@34996454 -- APPARENT DEADLOCK!!! Complete Status: 
            Managed Threads: 3
            Active Threads: 3
            Active Tasks: 
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@6730b844 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2f91ad49 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@507ac05 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
            Pending Tasks: 
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3aae7ed7
    Pool thread stack traces:
            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
                    java.net.SocketInputStream.socketRead0(Native Method)
                    java.net.SocketInputStream.read(SocketInputStream.java:150)
                    java.net.SocketInputStream.read(SocketInputStream.java:121)
                    oracle.net.ns.Packet.receive(Packet.java:300)
                    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
                    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
                    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
                    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
                    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
                    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
                    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
                    oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
                    oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
                    oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
                    oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
                    oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
                    oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
                    oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
                    com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
                    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
                    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
                    com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
                    com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
                    com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
                    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
                    java.net.SocketInputStream.socketRead0(Native Method)
                    java.net.SocketInputStream.read(SocketInputStream.java:150)
                    java.net.SocketInputStream.read(SocketInputStream.java:121)
                    oracle.net.ns.Packet.receive(Packet.java:300)
                    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
                    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
                    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
                    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
                    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
                    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
                    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
                    oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
                    oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
                    oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
                    oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
                    oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
                    oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
                    oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
                    com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
                    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
                    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
                    com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
                    com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
                    com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
                    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
            Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
                    java.net.SocketInputStream.socketRead0(Native Method)
                    java.net.SocketInputStream.read(SocketInputStream.java:150)
                    java.net.SocketInputStream.read(SocketInputStream.java:121)
                    oracle.net.ns.Packet.receive(Packet.java:300)
                    oracle.net.ns.DataPacket.receive(DataPacket.java:106)
                    oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
                    oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
                    oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
                    oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
                    oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
                    oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
                    oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
                    oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:380)
                    oracle.jdbc.driver.T4CTTIoauthenticate.doOAUTH(T4CTTIoauthenticate.java:760)
                    oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:401)
                    oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
                    oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
                    oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
                    oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
                    com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
                    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
                    com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
                    com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
                    com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
                    com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
                    com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
                    com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Related questions

Connection pooling options with JDBC: DBCP vs C3P0

What is the best connection pooling library available for Java/JDBC? I'm considering the 2 main candidates (free / open-source): Apache DBCP - http://commons.apache.org/dbcp/ C3P0 - http://sourceforge.net/projects/c3p0 I've read a lot about …

Read More
Spring JDBC connection pool best practices

I have a basic Spring JDBC application with a pretty basic configuration: <bean id="myDataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource"> <property name="driverClassName" value="oracle.jdbc.OracleDriver"/> <property name="url" value="jdbc:oracle:thin:@1.1.1.1:1521:…

Read More
JDBC Connection pooling using C3P0

Following is my helper class to get DB connection: I've used the C3P0 connection pooling as described here. public class DBConnection { private static DataSource dataSource; private static final String DRIVER_NAME; private static final String URL; private static final …

Read More