apache commons dbcp connection pool error: Timeout waiting for idle object in a Spring + Hibernate app using Spring Transactions

Dhwanit picture Dhwanit · Sep 26, 2013 · Viewed 26.5k times · Source

I have read through various Stackover flow Questions and contents on the web on similar problem. However, I couldnt find useful hints that would allow me to narrow down on my problem. Here is my usecase which results in this error.

2 entities Campus and Programs --> One-to-many relation from Campus to Program and One-to-one from Program to Campus.

i am trying to create multiple programs associated with campuses. Each insert will create a new program with same details and attach it to a different(unique) campus. eg. Java 101 Course offered at New York, San Francisco, Dallas, Chicago will create a new program for each campus.

There is no problem persisting single program. also there is no issue persisting a program for upto 46 campuses but this error shows up for the 47th campus.

Here is the config for my application:

Properties file:

datasource.classname=com.mysql.jdbc.Driver
datasource.url=jdbc:mysql://localhost:3306/edu
datasource.username=xxx
datasource.password=xxx123
datasource.initialsize=15
datasource.maxactive=50
datasource.maxidle=15
datasource.minidle=5
datasource.maxwait=10000
datasource.dialect=org.hibernate.dialect.MySQLDialect
datasource.validationquery =select 1
datasource.minEvictableIdleTimeMillis = 180000
datasource.timeBetweenEvictionRunsMillis = 180000

hibernate.batchsize=30

Here's how my spring-hibernate config looks like

<context:property-placeholder location="classpath:database.properties" order="1" ignore-unresolvable="true" />
<context:property-placeholder location="classpath:app.properties" order="2" ignore-unresolvable="true" />

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
    <property name="driverClassName" value="${datasource.classname}" />
    <property name="url" value="${datasource.url}" />
    <property name="username" value="${datasource.username}" />
    <property name="password" value="${datasource.password}" />
    <property name="initialSize" value="${datasource.initialsize}" />
    <property name="maxActive" value="${datasource.maxactive}" />
    <property name="maxIdle" value="${datasource.maxidle}" />
    <property name="minIdle" value="${datasource.minidle}" />
    <property name="maxWait" value="${datasource.maxwait}" />
    <property name="minEvictableIdleTimeMillis" value="${datasource.minEvictableIdleTimeMillis}" />
    <property name="timeBetweenEvictionRunsMillis" value="${datasource.timeBetweenEvictionRunsMillis}" />
    <property name="validationQuery" value="${datasource.validationquery}" />
    <property name="testOnBorrow" value="true" />
    <property name="removeAbandoned" value="true"/>
    <property name="removeAbandonedTimeout" value="60"/>
    <property name="logAbandoned" value="true"/>
</bean>

<bean id="sessionFactory"
    class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="hibernateProperties">
        <props>
            <prop key="hibernate.dialect">${datasource.dialect}</prop>
            <prop key="hibernate.show_sql">false</prop>
            <prop key="hibernate.format_sql">true</prop>
            <prop key="hibernate.batch_size">${hibernate.batchsize}</prop>
            <prop key="hibernate.current_session_context_class">org.springframework.orm.hibernate3.SpringSessionContext</prop>

            <prop key="hibernate.cache.use_second_level_cache">true</prop>
            <prop key="hibernate.cache.use_query_cache">true</prop>
            <prop key="hibernate.cache.provider_configuration_file_resource_path">/WEB-INF/ehcache-entity.xml</prop>
            <prop key="hibernate.cache.region.factory_class">net.sf.ehcache.hibernate.EhCacheRegionFactory</prop>
            <prop key="hibernate.generate_statistics">false</prop>
            <!-- <prop key="hibernate.connection.release_mode">auto</prop> -->
        </props>
    </property>

    <property name="packagesToScan">
        <list>
            <value>com.edapp.core</value>
            <value>com.edapp.data.engine</value>
            <value>com.edapp.service.engine</value>
        </list>
    </property>
</bean>

<bean id="transactionManager"
    class="org.springframework.orm.hibernate3.HibernateTransactionManager">
    <property name="sessionFactory" ref="sessionFactory" />
</bean>

<aop:config proxy-target-class="true"/> <!-- necessary to call methods on classes than proxies -->

<context:annotation-config /> 
<context:component-scan base-package="com.edapp" />

<!-- transaction settings -->

<tx:annotation-driven transaction-manager="transactionManager" />

Here's how my application flow looks like

ProgramController -> ProgramService -> ProgramDAO

Service class is annotated with: @Transactional(propagation=Propagation.REQUIRED, isolation=Isolation.READ_COMMITTED)

DAO class is annotated with: @Transactional(propagation=Propagation.MANDATORY, isolation=Isolation.READ_COMMITTED)

Here's the piece from Controller

List<String> campuses = Arrays.asList(gson.fromJson(campusesJSArray, String[].class));      
        if(campuses.size() > 0){
            List<Program> programList = new ArrayList<Program>();
            AreaOfStudy aos = this.areaOfStudyService.getById(areaOfStudyId);
            Concentration con = this.concentrationService.getById(concentrationId);
            for(String c : campuses){
                Long campusid = Long.parseLong(c);
                Program p = new Program();
                Campus campus = this.campusService.getById(campusid);
                if(campus != null){
                    System.out.println(campus.toString());
                    p.setName(name);
                    p.setCampus(campus);
                    p.setCode(code);
                    p.setLevel(level);
                    p.getCampus().getPrograms().add(p);
                    p.setAreaOfStudy(aos);
                    p.setConcentration(con);
                    p.setActive(true);
                }               
                programList.add(p);
            }
            ((ProgramServiceImpl)programService).saveOrUpdate(programList);

Here's a snippet from Service

    if(programList == null){
        log.error("ProgramList cannot be null");
        return null;
    }

    Map<Integer, String> errors = new HashMap<Integer, String>();
    log.info("Saving program list of size:"+programList.size());

    for(int i=0; i<programList.size();i++){
        try{
            this.saveOrUpdate(programList.get(i));
        }catch(HibernateException e){
            errors.put(i, "error");
        }
    }
    return errors;

Here's a snippet from the DAO:

@Transactional(isolation=Isolation.REPEATABLE_READ)
    public void create(final T entity) {
        if(entity == null){
            IllegalArgumentException e = new IllegalArgumentException();
            throw(e);
        }
        Session session = this.sessionFactory.getCurrentSession();
        try{
            session.persist(entity);
            session.flush();
        }catch(ConstraintViolationException cve){
            log.error("School with same code already exists "+ this.clazz.getName(),cve);
            throw cve;
        }catch(HibernateException e){
            log.error("Error persisting entity of type "+ this.clazz.getName(),e);
            throw new HibernateException(e);
        }finally{
            session.clear();
        }
    }

Batchsize = 30

@Transactional(isolation=Isolation.REPEATABLE_READ)
    public void create(List<T> entityList){
        if(entityList == null){
            IllegalArgumentException e = new IllegalArgumentException();
            throw(e);
        }
        Session session = this.sessionFactory.getCurrentSession();
          try{
            for(int i=0;i<entityList.size();i++){
                T entity = entityList.get(i);

                if(entity == null){
                    log.error("List "+ this.clazz.getName() + " of cannot contain null");
                    throw new NullPointerException("List "+ this.clazz.getName() + " of cannot contain null");
                }
                session.persist(entity);
                if(i% this.batchSize == 0){
                    session.flush();
                    session.clear();
                }
            }
        }catch(HibernateException e){
            log.error("Error persisting entity of type "+ this.clazz.getName(),e);
            throw new HibernateException(e);
        }finally{
            session.flush();
            session.clear();
        }
    }

tried using both methods to persist but same outcome.


Here is the full stack trace

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:932)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:801)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:233)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

root cause

org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
    org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:597)
    org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
    org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:329)
    org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
    com.tr.leadgen.service.engine.CampusServiceImpl$$EnhancerByCGLIB$$68d579ad.getById(<generated>)
    com.tr.leadgen.web.edu.controllers.ProgramController.add(ProgramController.java:74)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:920)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:801)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:233)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

root cause

org.hibernate.exception.GenericJDBCException: Cannot open connection
    org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
    org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
    org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
    org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
    org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
    org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
    org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
    org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
    org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
    org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:556)
    org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
    org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:329)
    org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
    com.tr.leadgen.service.engine.CampusServiceImpl$$EnhancerByCGLIB$$68d579ad.getById(<generated>)
    com.tr.leadgen.web.edu.controllers.ProgramController.add(ProgramController.java:74)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:920)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:801)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:233)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

root cause

org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:83)
    org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
    org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
    org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
    org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
    org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
    org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:556)
    org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
    org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:329)
    org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
    com.tr.leadgen.service.engine.CampusServiceImpl$$EnhancerByCGLIB$$68d579ad.getById(<generated>)
    com.tr.leadgen.web.edu.controllers.ProgramController.add(ProgramController.java:74)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:920)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:801)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:233)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

root cause

java.util.NoSuchElementException: Timeout waiting for idle object
    org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
    org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
    org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
    org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:83)
    org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
    org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
    org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
    org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
    org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
    org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:556)
    org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
    org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:329)
    org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
    com.tr.leadgen.service.engine.CampusServiceImpl$$EnhancerByCGLIB$$68d579ad.getById(<generated>)
    com.tr.leadgen.web.edu.controllers.ProgramController.add(ProgramController.java:74)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:219)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:920)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:827)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:801)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:233)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

This is the only way I get session from hibernate:

@Transactional(propagation=Propagation.NOT_SUPPORTED)
    protected Session getCurrentSession(){
        if(this.sessionFactory == null){
            log.error("SessionFactory is null");
        }
        return this.sessionFactory.getCurrentSession();

}

Also per my app logs, code execution does not make it to service layer. It throws an error while looping in the controller to populate the list of program entites (refer controller snippet)

I would be thankful if anyone can point me in the right direction.

Answer

gerrytan picture gerrytan · Sep 26, 2013

I see you had a pool setting which validates connection on borrow, and from the stack trace I can see this validation failed (seem to timeout). There are 2 possible causes I can think of:

  1. Network connection between your app and db server was broken. This could be due to connection lost, firewall setting change, stale DNS entries, db server suddenly dies, or even router had a setting to kill suspicious tcp socket (happened to us once)

  2. The pool run out of available connection. This is rather unlikely because I'd assume the stack trace would give more hint, but it's worth monitoring your pool and check the number of available conn at the time when problem occur