WinDbg x64: Cannot debug a crash dump - failed to load data access DLL

debugging windbg dump
net_prog picture net_prog · Aug 17, 2011 · Viewed 37.1k times · Source

I attached WinDbg to a running process and had the process crashed (I have a separate question re. that case). Once the program crashed, WinDbg stopped and allowed me to debug the program. I took a crash dump for further investigation with a command ".dump /ma".

The program was compiled as "Any CPU" and I used WinDbg x64 to take the dump. Now I open WinDbg x64 on the same computer again and open the crash dump. Here is what it says:

Loading Dump File [C:\crashdump.dmp]
User Mini Dump File with Full Memory: Only application data is available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00)
System Uptime: 17 days 0:54:39.021
Process Uptime: 12 days 14:01:31.000
................................................................
...............................................................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1be0.b78): Access violation - code c0000005 (first/second chance not available)
*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
clr!WKS::gc_heap::find_first_object+0x92:
000007fe`ea129a1d f70100000080    test    dword ptr [rcx],80000000h ds:00000000`00003d80=????????

Then I try to load SOS by ".load sos clr" and it errors in:

The call to LoadLibrary(sos clr) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

Trying with ".loadby sos clr" and it works. Now I want to see the stack with "!clrstack" and stick here:

Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is 
                in the version directory
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

I tried ".symfix" and ".reload":

0:027> .symfix
0:027> .reload
..................*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
..............................................
...............................................................

Stuck. At the same time when the process is running under WinDgb I can pause the execution, load SOS and execute "!clrstack" command successfully.

Any ideas? Thank you.

UPDATE - Followed the steps provided in the second answer, still doesn't work.

1) My Symbol Path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*

2) CLR loaded: 4.0.30319.237: 

0:027> lm v clr
Unknown option 'r'
start             end                 module name
00000000`77b60000 00000000`77d09000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb
    Loaded symbol image file: ntdll.dll
    Image path: C:\Windows\System32\ntdll.dll
    Image name: ntdll.dll
    Timestamp:        Sat Nov 20 13:11:21 2010 (4CE7C8F9)
    CheckSum:         001B55EA
    ImageSize:        001A9000
    File version:     6.1.7601.17514
    Product version:  6.1.7601.17514
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® Windows® Operating System
    InternalName:     ntdll.dll
    OriginalFilename: ntdll.dll
    ProductVersion:   6.1.7601.17514
    FileVersion:      6.1.7601.17514 (win7sp1_rtm.101119-1850)
    FileDescription:  NT Layer DLL
    LegalCopyright:   © Microsoft Corporation. All rights reserved.
000007fe`e9fb0000 000007fe`ea915000   clr      # (pdb symbols)          c:\symbols\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb
    Loaded symbol image file: clr.dll
    Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    Image name: clr.dll
    Timestamp:        Tue May 17 09:35:10 2011 (4DD2333E)
    CheckSum:         00967144
    ImageSize:        00965000
    File version:     4.0.30319.237
    Product version:  4.0.30319.237
    File flags:       8 (Mask 3F) Private
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     clr.dll
    OriginalFilename: clr.dll
    ProductVersion:   4.0.30319.235
    FileVersion:      4.0.30319.235 (RTMGDR.030319-2300)
    PrivateBuild:     DDBLD240
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail

3) "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll" has version 4.0.30319.239

4) I found that when I load the dump into WinDbg it loads the correct "mscordacwks.dll" from the web, thus in the folder "C:\symbols\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000" I have the file "mscordacwks_AMD64_AMD64_4.0.30319.237.dll".

5) 

0:027> .cordll -ve -u -l
CLR DLL status: No load attempts

6)

0:027> !sym noisy
noisy mode - symbol prompts on
0:027> .restart

Loading Dump File [C:\crashdump.dmp]
User Mini Dump File with Full Memory: Only application data is available

DBGHELP: Symbol Search Path: srv*;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00)
System Uptime: 17 days 0:54:39.021
Process Uptime: 12 days 14:01:31.000
................................................................
...............................................................
DBGHELP: ntdll - public symbols  
         C:\Program Files\Debugging Tools for Windows (x64)\sym\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(1be0.b78): Access violation - code c0000005 (first/second chance not available)
*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll
DBGHELP: clr - public symbols  
         C:\Program Files\Debugging Tools for Windows (x64)\sym\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb
clr!WKS::gc_heap::find_first_object+0x92:
000007fe`ea129a1d f70100000080    test    dword ptr [rcx],80000000h ds:00000000`00003d80=????????

7)

0:027> !clrstack
SYMSRV:  C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll not found
SYMSRV:  mscordacwks_AMD64_AMD64_4.0.30319.237.dll from http://msdl.microsoft.com/download/symbols: 502892 bytes - copied         
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll cached to C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll - OK
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is 
                in the version directory
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to clr.dll as well.

Answer

Pete picture Pete · Oct 24, 2011

I hit this regularly when debugging with minidumps from the site. Quite how it's happened in your case I'm not sure. Usually, it happens when the version of CLR that was loaded when the dump was taken is not available on your debugging machine. In your case, they're the same machine, so it should all probably just work. I'm sure there will be others who can explain exactly why it isn't.

In the meantime, here's what I do with my site dumps. Windbg is looking for "the right version" of mscordacwks.dll. So we give it that version and tell it where to look for it.

First - if I spoof all of this, by deleting mscordacwks.dll, windbg goes off and loads it from the Microsoft symbol server, so do make sure your symbols are set up correctly to download symbols from the Microsoft symbol server and give it another go.

Now - assuming that didn't work, check exactly which version is the "right version". List the module info with "lm v clr" and check your CLR version that is ACTUALLY loaded. Mine is 4.0.30319.239. Ok - now find that version of mscordacwks.dll. Let's assume it can be found in the normal .NET framework installation on your machine (C:\Windows\Microsoft.NET\Framework64\v4.0.30319). Do check the version matches exactly (right-click, properties etc)! Take it and put it in a safe place (I use D:\Symbols\_Images). Follow the instructions that windbg gave you on renaming the file. mscordacwks_.dll would be mscordacwks_AMD64_AMD64_4.0.30319.239.dll.

Now set up your executable image path (".exepath D:\Symbols\_Images") so windbg knows where you've put it.

You've now got "the right version of mscordacwks", and renamed it so that Windbg knows what it's looking for, and told it where you've put it.

If that STILL isn't working, then try ".cordll -ve -u -l" and also "!sym noisy" to turn on verbose logging of both the cordll load and the symbol server, then try !CLRStack again. Maybe the output of those two commands will tell you exactly what it's trying to load and you can figure out why it won't do it...

Related questions

WinDbg and SoS, how do I print/dump a large string?

I am debugging a hangdump coming from a production server using WinDbg with the SoS extension. There is a string parameter in one of the stacks, that I need to know the value of. However, it is a rather large …

Read More
How to set up symbols in WinDbg?

I am using Debugging Tools for Windows and I get the following error message when starting WinDbg / cdb or ntsd: Symbol search path is: *** Invalid *** **************************************************************************** * Symbol loading may be unreliable without a symbol search path. * * Use .symfix to have the …

Read More
ERROR: Symbol file could not be found. windbg.exe

I downloaded the symbols from http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx the package (Windows 7 RTM x86 retail symbols, all languages) but it still gives me the same error! Done .sympath but still not working .sympath C:\inetpub\…

Read More