We have a production website with a mixture of classic ASP, inline .Net and compiled .Net that talks to a SQL Server instance where both are VMs on the same physical box and everything there works great. In order to do some testing with new features I created a QA version of the site and have it setup to connect to a database on a VM hosted on another physical server. They are different networks (DMZ/internal), but all in the same rack on gigabit so connection speeds shouldn't be an issue.
The problem we're running into is the QA version will work for a bit and then suddenly dies until I restart the app pool and the site. It's not a load issue because it's just me and two other people testing this process and I've confirmed the server where the QA database is at doesn't have a resource issue (16GB RAM, 3.2GB in use and processor utilization hovers in the 2% range). The specific error (shown below) occurs with all variants of the pages - classic ASP, inline .Net and the compiled .Net pages. Based on all the research I've done, I keep getting pointed back to it being a firewall issue, but we even opened all ports between the two servers and it still happens. The strangest part is when we enable tracing on our ASA (5515-X), it sees hardly any traffic and sometimes the site kicks back that error immediately while other times it takes 30+ seconds. The pages in question aren't doing any intense lookups (one page that dies is grabbing a total of about 60 records) as well. Here is the full error/stack trace:
[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgement. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=21008; handshake=12; ]
System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) +6749670
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) +815
System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) +817
System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() +344
System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() +48
System.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean& marsCapable) +126
System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover) +693
System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover) +219
System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout) +6777754
System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance) +6778255
System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData) +878
System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions) +1162
System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions) +72
System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) +6781425
System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) +103
System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) +2105
System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) +116
System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) +1089
System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) +6785863
System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry) +233
System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) +278
System.Data.SqlClient.SqlConnection.Open() +239
System.Web.UI.Control.LoadRecursive() +71
System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +3178
I'd show a code block, but since all pages start dying once one does it wouldn't do much good. This all works great in production so the only difference is that the VMs are on different physical hosts, but given that it's all new hardware there's no reason this should have issues.
I've also come across multiple things on here where using exec sp_updatestats
and dbcc freeproccache
, however neither of those seemed to make a difference.
I'm completely lost so any suggestions?
Had a similar problem with SQL Express 2014, turned out to be a Windows Firewall problem, some of our Servers have SQL 2005/2008 on them, all with the Windows Firewall disabled.
Made the change on my PC and it worked perfectly afterwards!
Thanks
Brendan