We have an application running on JBoss 4.2.3, using Spring 2.5.2 and Hibernate 3.2.6.ga. This is running on Linux JEE01 2.6.16.60-0.54.5-smp, using its own user. Writing to a Oracle 10G database on another machine.
We're using a standard view -> service -> dao layering. Where each dao is annotated with @Repository.
This is all running 24/7 without many problems, but every several days and sometimes a couple of times in one day the whole system goes into a bad state where nothing can be written to the database anymore. These stacktraces appear in the logs:
org.springframework.dao.InvalidDataAccessApiUsageException: Write operations are not
allowed in read-only mode (FlushMode.NEVER/MANUAL):
Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction
definition.
We scanned the complete system, and there is one place in the system where the flushmode is temporarely set to MANUAL after which a finally block set its back to its original value. This is because we do not want to flush the state to the database before this query runs. So we can't change this very easily. The normal FlushMode is set to AUTO and on several places we temporarily set it to COMMIT and switching it back to the default again.
Only a server restart restores the system back to working order.
The question is: why does the system set all transactions to readonly/manual flush mode? I googled this but could not find a solution.
This is our spring and hibernate configuration (only relevants part showing):
<bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
<property name="dataSource">
<ref bean="datasourceName" />
</property>
<property name="configLocation">
<value>classpath:hibernate.cfg.xml</value>
</property>
</bean>
<bean id="hibernateInterceptor"
class="org.springframework.orm.hibernate3.HibernateInterceptor">
<property name="sessionFactory" ref="sessionFactory" />
</bean>
<tx:annotation-driven transaction-manager="txManager" />
<tx:advice id="txAdvice" transaction-manager="txManager" >
<!-- the transactional semantics... -->
<tx:attributes >
<!-- all methods starting with 'get' are read-only -->
<tx:method name="approve*" read-only="false"
propagation="REQUIRED" rollback-for="java.lang.Exception" />
<tx:method name="update*" read-only="false"
propagation="REQUIRED" rollback-for="java.lang.Exception"/>
<tx:method name="save*" read-only="false"
propagation="REQUIRED" rollback-for="java.lang.Exception"/>
<tx:method name="delete*" read-only="false"
propagation="REQUIRED" rollback-for="java.lang.Exception" />
<!-- other methods use the default transaction settings (see below) -->
<tx:method name="*" read-only="true" propagation="REQUIRED" />
</tx:attributes>
</tx:advice>
<aop:config>
<aop:pointcut id="serviceMethods"
expression="execution(* com.myapplication.service.*.*(..))" />
<aop:advisor advice-ref="txAdvice"
pointcut-ref="serviceMethods" />
</aop:config>
<bean id="txManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager" >
<property name="sessionFactory" ref="sessionFactory" />
</bean>
-- end of spring config --
-- hibernate configuation --
<hibernate-configuration>
<session-factory name="">
<property name="dialect">org.hibernate.dialect.Oracle10gDialect</property>
<property name="show_sql">false</property>
<property name="use_outer_join">false</property>
<property name="hibernate.cache.use_query_cache">true</property>
<property name="hibernate.cache.use_second_level_cache">true</property>
<property name="cache.provider_class">org.hibernate.cache.EhCacheProvider</property>
<property name="hibernate.connection.SetBigStringTryClob">true</property>
<property name="hibernate.jdbc.batch_size">0</property>
</session-factory>
<mapping ----/>
</hibernate-configuration>
This is the stacktrace:
org.springframework.dao.InvalidDataAccessApiUsageException: Write operations are not allowed in read-only mode (FlushMode.NEVER/MANUAL): Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction definition.
at org.springframework.orm.hibernate3.HibernateTemplate.checkWriteOperationAllowed(HibernateTemplate.java:1137)
at org.springframework.orm.hibernate3.HibernateTemplate$16.doInHibernate(HibernateTemplate.java:701)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:374)
at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:699)
at nl.company.myapp.dao.impl.GenericDAOImpl.save(GenericDAOImpl.java:94)
at nl.company.myapp.dao.impl.CallDAOImpl.save(CallDAOImpl.java:266)
at nl.company.myapp.dao.impl.CallDAOImpl.save(CallDAOImpl.java:47)
at nl.company.myapp.service.impl.CallServiceImpl.saveCall(CallServiceImpl.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy142.saveCall(Unknown Source)
at nl.company.myapp.view.bean.call.CallDetailBean.doSave(CallDetailBean.java:319)
at nl.company.myapp.view.bean.EditModeAwareBean.save(EditModeAwareBean.java:151)
at sun.reflect.GeneratedMethodAccessor472.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.apache.el.parser.AstValue.invoke(AstValue.java:131)
at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
at com.sun.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:68)
at org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
at org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:190)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:458)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:763)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at nl.company.myapp.view.audit.AuditFilter.doFilter(AuditFilter.java:88)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._invokeDoFilter(TrinidadFilterImpl.java:238)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:195)
at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:138)
at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:366)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
at java.lang.Thread.run(Thread.java:595)
This all works fine
This exception comes from the following code in Spring's HibernateTemplate class:
protected void checkWriteOperationAllowed(Session session) throws InvalidDataAccessApiUsageException {
if (isCheckWriteOperations() && getFlushMode() != FLUSH_EAGER &&
session.getFlushMode().lessThan(FlushMode.COMMIT)) {
throw new InvalidDataAccessApiUsageException(
"Write operations are not allowed in read-only mode (FlushMode.NEVER/MANUAL): "+
"Turn your Session into FlushMode.COMMIT/AUTO or remove 'readOnly' marker from transaction definition.");
}
}
The rationale for this check is explained as:
This is a new consistency check introduced in Spring 1.1.
Invoking HibernateTemplate's save/update/delete methods on a Spring-managed Session in FlushMode.NEVER is potentially dangerous: It means that you are:
either doing this in a Spring-managed read-only transaction, which will never flush the Hibernate Session, i.e. never flush your save/update/delete calls. The new check makes you aware that you won't persist your changes in that situation.
or working with some other thread-bound Session in FlushMode.NEVER, for example the OpenSessionInViewFilter. If you're overriding closeSession there to flush after view rendering, you should override getSession too, setting the Session to FlushMode.AUTO.
I strongly recommend against the latter, though. If you're using OpenSessionInViewFilter, combine it with middle tier transactions rather than let the filter flush at request completion.
Does any of this ring a bell?
It is possible that there is some bug in the your code or in Spring ORM. To disable this check you can call setCheckWriteOperations(false)
.