XAException : 2-Phase commit Rollback failing due to XA Exception

Andy picture Andy · Mar 22, 2012 · Viewed 12k times · Source

Ok a seemingly simple implementaton of 2-Phase commit is running into issues. My setup is 2 databases on a DB2 server,an application hosted on Websphere 6.1. Transaction Management handled by Atomikos. Database connection through jndi. I begin the UserTransaction, execute an update on DB1, lose connection to the DBs while update to DB2 to get a StaleConnectionException which invokes a rollback. Now ideally, the TM should execute rollback for the transaction. However, I get an XAException as below:

[3/22/12 11:52:21:022 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_0.txt

[3/22/12 11:52:21:053 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_0.txt
[3/22/12 11:52:21:053 EDT] 00000023 WSRdbXaResour E   DSRA0304E:  XAException occurred. XAException contents and details are: 
The DB2 Error message is : [jcc][t4][10335][11391][3.53.70] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003
The DB2 Error code is    : -4470
The DB2 SQLState is      : 08003
.
[3/22/12 11:52:21:053 EDT] 00000023 WSRdbXaResour E   DSRA0302E:  XAException occurred.  Error code is: XAER_RMFAIL (-7).  Exception is: [jcc][t4][10401][12066][3.53.70] XA exception: XAER_RMFAIL : [jcc][t4][10335][11391][3.53.70] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003 ERRORCODE=-4228, SQLSTATE=null
[3/22/12 11:52:21:085 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_1.txt
[3/22/12 11:52:21:100 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_1.txt
[3/22/12 11:52:21:116 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_2.txt
[3/22/12 11:52:21:163 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_2.txt
[3/22/12 11:52:21:194 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_3.txt
[3/22/12 11:52:21:225 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_3.txt
[3/22/12 11:52:21:256 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_4.txt
[3/22/12 11:52:21:288 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_4.txt
[3/22/12 11:52:21:288 EDT] 00000023 WSRdbXaResour E   DSRA0304E:  XAException occurred. XAException contents and details are: 
The DB2 Error message is : [jcc][t4][2030][11211][3.53.70] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: T4Agent.sendRequest().  Message: Connection reset by peer: socket write error. ERRORCODE=-4499, SQLSTATE=08001
The DB2 Error code is    : -4499
The DB2 SQLState is      : 08001
.
[3/22/12 11:52:21:288 EDT] 00000023 WSRdbXaResour E   DSRA0302E:  XAException occurred.  Error code is: XA_RETRY (4).  Exception is: [jcc][t4][10401][12066][3.53.70] XA exception: XA_RETRY : [jcc][t4][2030][11211][3.53.70] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: T4Agent.sendRequest().  Message: Connection reset by peer: socket write error. ERRORCODE=-4499, SQLSTATE=08001 ERRORCODE=-4228, SQLSTATE=null
[3/22/12 11:52:21:288 EDT] 00000023 ConnectionEve A   J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc/myDatasource1.  The exception which was received is com.ibm.db2.jcc.b.po: [jcc][t4][10401][12066][3.53.70] XA exception: XA_RETRY : [jcc][t4][2030][11211][3.53.70] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: T4Agent.sendRequest().  Message: Connection reset by peer: socket write error. ERRORCODE=-4499, SQLSTATE=08001 ERRORCODE=-4228, SQLSTATE=null:com.ibm.db2.jcc.b.tn: [jcc][t4][2030][11211][3.53.70] A communication error occurred during operations on the connection's underlying socket, socket input stream, 
or socket output stream.  Error location: T4Agent.sendRequest().  Message: Connection reset by peer: socket write error. ERRORCODE=-4499, SQLSTATE=08001:java.net.SocketException: Connection reset by peer: socket write error
[3/22/12 11:52:21:319 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_5.txt
[3/22/12 11:52:21:335 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_5.txt
[3/22/12 11:52:21:350 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_6.txt
[3/22/12 11:52:21:397 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_6.txt
[3/22/12 11:52:21:413 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_7.txt
[3/22/12 11:52:21:444 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_7.txt
[3/22/12 11:52:21:522 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl open FFDC0009I: FFDC opened incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_8.txt
[3/22/12 11:52:21:553 EDT] 00000023 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file C:\Program Files\IBM\SDP\runtimes\base_v61\profiles\was61profile1\logs\ffdc\server1_00000023_12.03.22_11.52.21_8.txt
[3/22/12 11:53:16:804 EDT] 00000023 SystemOut     O Transaction Failed
[3/22/12 11:53:21:522 EDT] 0000000f InternalDB2Un I   DSRA8203I: Database product name : DB2/NT
[3/22/12 11:53:21:522 EDT] 0000000f InternalDB2Un I   DSRA8204I: Database product version : SQL09056
[3/22/12 11:53:21:522 EDT] 0000000f InternalDB2Un I   DSRA8205I: JDBC driver name  : IBM DB2 JDBC Universal Driver Architecture
[3/22/12 11:53:21:522 EDT] 0000000f InternalDB2Un I   DSRA8206I: JDBC driver version  : 3.53.70
[3/22/12 11:53:21:522 EDT] 0000000f InternalDB2Un I   DSRA8212I: DataStoreHelper name is: com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper@150a150a.
[3/22/12 11:53:21:522 EDT] 0000000f WSRdbDataSour I   DSRA8208I: JDBC driver type  : 4
[3/22/12 11:53:21:585 EDT] 0000000f WSRdbXaResour E   DSRA0304E:  XAException occurred. XAException contents and details are: "".
[3/22/12 11:53:21:585 EDT] 0000000f WSRdbXaResour E   DSRA0302E:  XAException occurred.  Error code is: XAER_NOTA (-4).  Exception is: [jcc][t4][10401][12066][3.53.70] XA exception: XAER_NOTA ERRORCODE=-4228, SQLSTATE=null

The DB1 table on which update was performed get LOCKED after this. I have tried the following till now but still unable to perform rollback:

  1. query is using uncommited reads ('ur')
  2. Connection auto commit is set to false and IsolationLevel to *TRANSACTION_READ_UNCOMMITTED*
  3. Increased the various timeout values for Datasource and Transaction Service.
  4. Connection pool minimum connections set to 0, Purge Policy: EntirePool
  5. StaleConnectionException is being caught and UserTransaction.rollback() called from the catch block.

How do i use 2-phase commit and XAtransactions to recover from a DB crash? Any pointers to what should be done for this?

Answer