Troubleshooting .NET "Fatal Execution Engine Error"

Question

Summary:

I periodically get a .NET Fatal Execution Engine Error on an application which I cannot seem to debug. The dialog that comes up only offers to close the program or send information about the error to Microsoft. I've tried looking at the more detailed information but I don't know how to make use of it.

Error:

The error is visible in Event Viewer under Applications and is as follows:

.NET Runtime version 2.0.50727.3607 - Fatal Execution Engine Error (7A09795E) (80131506)

The computer running it is Windows XP Professional SP 3. (Intel Core2Quad Q6600 2.4GHz w/ 2.0 GB of RAM) Other .NET-based projects that lack multi-threaded downloading (see below) seem to run just fine.

Application:

The application is written in C#/.NET 3.5 using VS2008, and installed via a setup project.

The app is multi-threaded and downloads data from multiple web servers using System.Net.HttpWebRequest and its methods. I've determined that the .NET error has something to do with either threading or HttpWebRequest but I haven't been able to get any closer as this particular error seems impossible to debug.

I've tried handling errors on many levels, including the following in Program.cs:

// handle UI thread exceptions
Application.ThreadException += Application_ThreadException;

// handle non-UI thread exceptions
AppDomain.CurrentDomain.UnhandledException += CurrentDomain_UnhandledException;

Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);

// force all windows forms errors to go through our handler
Application.SetUnhandledExceptionMode(UnhandledExceptionMode.CatchException);

More Notes and What I've Tried...

• Installed Visual Studio 2008 on the target machine and tried running in debug mode, but the error still occurs, with no hint as to where in source code it occurred.
• When running the program from its installed version (Release) the error occurs more frequently, usually within minutes of launching the application. When running the program in debug mode inside of VS2008, it can run for hours or days before generating the error.
• Reinstalled .NET 3.5 and made sure all updates are applied.
• Broke random cubicle objects in frustration.
• Rewritten parts of code that deal with threading and downloading in attempts to catch and log exceptions, though logging seemed to aggravate the problem (and never provided any data).

Question:

What steps can I take to troubleshoot or debug this kind of error? Memory dumps and the like seem to be the next step, but I'm not experienced at interpreting them. Perhaps there's something more I can do in the code to try and catch errors... It would be nice if the "Fatal Execution Engine Error" was more informative, but internet searches have only told me that it's a common error for a lot of .NET-related items.

Answer 1

Well, you've got a Big Problem. That exception is raised by the CLR when it detects that the garbage collected heap integrity is compromised. Heap corruption, the bane of any programmer that ever wrote code in an unmanaged language like C or C++.

Those languages make it very easy to corrupt the heap, all it takes is to write past the end of an array that's allocated on the heap. Or using memory after it has been released. Or having a bad value for a pointer. The kind of bugz that managed code was invented to solve.

But you are using managed code, judging from your question. Well, mostly, your code is managed. But you are executing lots of unmanaged code. All the low-level code that actually makes a HttpWebRequest work is unmanaged. And so is the CLR, it was written in C++ so is technically just as likely to corrupt the heap. But after over four thousand revisions of it, and millions of programs using it, the odds that it still suffers from heap cooties are very small.

The same isn't true for all the other unmanaged code that wants a piece of HttpWebRequest. The code you don't know about because you didn't write it and isn't documented by Microsoft. Your firewall. Your virus scanner. Your company's Internet usage monitor. Lord knows whose "download accelerator".

Isolate the problem, assume it is neither your code nor Microsoft's code that causes the problem. Assume it is environmental first and get rid of the crapware.

For an epic environmental FEEE story, read this thread.