dbo.TempGetStateItemExclusive3 called repeatedly
I am investigating a problem with our website that uses SQL server to manage sessions. The website is asp.net webforms based around the sitecore CMS. We have the same code in various environments e.g. QA, staging, and production.
In production, what we are seeing is, periodically, we get a rapidly rising CPU usage that does not correlate in any way to traffic to the server. Along with this cpu spike, we are seeing a corresponding spike in network I/O.
Our monitoring software does not differentiate between traffic out to the internet and traffic to the DB server; however, what we are seeing on the DB server is literally hundreds of calls per second to dbo.TempGetStateItemExclusive3 in the asp session database, all for the same session id, and no corresponding amount of page requests coming into the web servers.
With the same code and config, we simply do not see this behaviour for other environments. We also do not see it for other session ids, just this one specific one.
Deleting the row from the database simply results in it being recreated with the same session id.
UPDATE
I have found this error in the event log:
Violation of PRIMARY KEY constraint 'PK__ASPState__C9F49290145C0A3F'. Cannot insert duplicate key in object 'dbo.ASPStateTempSessions'. The duplicate key value is (sessionidwiththeproblem). The statement has been terminated.
Stack trace:
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean\ breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand\ cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler,\ TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds,\ RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior,\ RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior,\ RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult\ result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at System.Web.SessionState.SqlSessionStateStore.SqlExecuteNonQueryWithRetry(SqlCommand\ cmd, Boolean ignoreInsertPKException, String id)
Anyone any ideas how a duplicate session id could possibly be attempted to be created?
Answer
We've faced a similar problem having the following configuration:
- IIS 7.5
- .NET Framework 4.0
- Windows 2008 (on both the IIS and the DB server)
- sessions managed by the ASPState database
The problem was that sometimes some of the sessions remained locked into the ASPState database, resulting in hundreds of calls per second to dbo.TempGetStateItemExclusive3 for each of the locked sessions.
The CPU on the IIS server would eventually go up with the number of locked sessions. A temporary solution was to recycle the application pool.
Going further and enabling the Tracing on the IIS server(s) and then analyzing the traces we noticed that whenever there was a problem (i.e. network connectivity problem that caused a 500 Internal server error) in the EXECUTE_REQUEST_HANDLER module, the next module which is RELEASE_REQUEST_STATE (and should unlock the session) was not be executed. Thus the session remained locked.
It turned up to be a bug from IIS and we fixed it by changing the value of the uploadReadAheadSize to 0 in the web.config:
<system.webServer>
<serverRuntime uploadReadAheadSize="0" />
</system.webServer>
The UploadReadAheadSize property establishes the number of bytes a Web server will read into a buffer and pass to an ISAPI extension. This occurs once per client request.