It appears that there was a problem at some point that folks tracked down that caused authentication tickets to be marked invalid on a Webfarm when the servers had different mixes of patches.
UNFORTUNATELY, it seems that there is STILL a problem even when the web servers have identical patches on them.
My two servers:
Yet when a user transitions from one web server in the farm to the other web server the underlying microsoft code invalidates the token. Here's the Event log entry:
Event code: 4005
Event message: Forms authentication failed for the request. Reason: The ticket supplied was invalid.
Event time: 4/10/2012 2:42:20 PM
Event time (UTC): 4/10/2012 6:42:20 PM
Event ID: 92eedee52ede49239fd063fe5609d858
Event sequence: 2
Event occurrence: 1
Event detail code: 50201
Application information:
Application domain: /LM/W3SVC/2000/ROOT-1-129785553216092727
Trust level: Full
Application Virtual Path: /
Application Path: C:\inetpub\wwwroot\testportal.mydomain.com\
Machine name: WEB02
Process information:
Process ID: 1428
Process name: w3wp.exe
Account name: IIS APPPOOL\testportal.mydomain.com
Request information:
Request URL: http://testportal.mydomain.com/Interface.aspx
Request path: /Interface.aspx
User host address: *************
User:
Is authenticated: False
Authentication Type:
Thread account name: IIS APPPOOL\testportal.mydomain.com
Name to authenticate:
Custom event details:
I've either got to figure this out or write microsoft's authentication out of my system (which I really don't have time to do).
Here are the hotfixes applied IDENTICALLY to both servers which are:
Windows Server 2008 R2 x64-based
KB981391,KB981392,KB977236,KB981111,KB977238,KB977239,KB981390,KB2305420,KB2386667,KB2393802,KB2425227,KB2475792,KB2476490,KB2478662,KB2479628,KB2482017,KB2484033,KB2485376,KB2487426,KB2488113,KB2492386,KB2503665,KB2505438,KB2506014,KB2506212,KB2506928,KB2507618,KB2507938,KB2508272,KB2509553,KB2510531,KB2511250,KB2511455,KB2515325,KB2518869,KB2522422,KB2524375,KB2529073,KB2530548,KB2533552,KB2533623,KB2534366,KB2536275,KB2536276,KB2539635,KB2541014,KB2544521,KB2544893,KB2545698,KB2547666,KB2552343,KB2555917,KB2556532,KB2560656,KB2563227,KB2564958,KB2567680,KB2570947,KB2572077,KB2584146,KB2585542,KB2588516,KB2598845,KB2603229,KB2607047,KB2607576,KB2608658,KB2618444,KB2618451,KB2620704,KB2620712,KB2621440,KB2631813,KB2632503,KB2633873,KB2633952,KB2636573,KB2639308,KB2639417,KB2640148,KB2641653,KB2641690,KB2643719,KB2644615,KB2645640,KB2647516,KB2647518,KB2654428,KB2656356,KB2660075,KB2660465,KB2665364,KB2667402,KB958488,KB976902,KB976932,KB982018
Per request, here's the authentication section of my web.config:
<authentication mode="Forms">
<forms loginUrl="Login.aspx" name=".ASPXFORMSAUTH" cookieless="AutoDetect" timeout="120" slidingExpiration="true">
</forms>
</authentication>
I think I have finally resolved this issue. I'm not exactly sure why this change fixes the problem but what I did to resolve it was to put the machineKey definition directly in the web.config file for my application. Apparently either I don't understand how to use IIS Manager to set up the machine keys properly or there is some issue with the IIS Manager.
So to solve the problem I put an entry directly in my web.config that looks something like this:
<machineKey validation="SHA1" validationKey="-a-validation-key-" decryption="Auto" decryptionKey="-a-decryption-key-"/>
This article kind of led me in the correct path to solve this issue: