Windbg crash dump analysis
I'm having a hard time getting any meaningful information from a crash dump I created with ProcDump, but I'm pretty sure it's relevant to a seemingly random crash I've been having.
I have a VB6 application running on Windows 7 64-bit. Every once in a while, it crashes, leaving an entry in the error log that faults ntdll.dll but gives no more information than that. So, I have been running the process with SysInternals' ProcDump running to automatically create crash dumps for me.
I've been unable to re-create the crash in-house, so I was pretty sure that if I had a dump at all, it would tell me what the issue was. However, after running for most of a day, I see that ProcDump wrote several dumps already, although the program is still running fine. It does seem to point to issues with ntdll.dll, but I have no idea where to begin applying a fix for this.
Running !analyze -v on one of the dumps gives me the following:
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 000007c8
PROCESS_NAME: application.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: application.exe
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT_AFTER_CALL
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL
LAST_CONTROL_TRANSFER: from 7754431f to 7752014d
STACK_TEXT:
0382fdf4 7754431f 00000005 035e62c8 00000001 ntdll!ZwWaitForMultipleObjects+0x15
0382ff88 74cd339a 00000000 0382ffd4 77539ed2 ntdll!TppWaiterpThread+0x33d
0382ff94 77539ed2 035e6298 74e2a30c 00000000 kernel32!BaseThreadInitThunk+0xe
0382ffd4 77539ea5 775441f3 035e6298 00000000 ntdll!__RtlUserThreadStart+0x70
0382ffec 00000000 775441f3 035e6298 00000000 ntdll!_RtlUserThreadStart+0x1b
STACK_COMMAND: ~0s; .ecxr ; kb
FOLLOWUP_IP:
ntdll!ZwWaitForMultipleObjects+15
7752014d 83c404 add esp,4
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!ZwWaitForMultipleObjects+15
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ba58
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_AFTER_CALL_80000003_ntdll.dll!ZwWaitForMultipleObjects
BUCKET_ID: APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL_ntdll!ZwWaitForMultipleObjects+15
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/BlackJack_exe/1_5_0_0/50227d4e/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1
Followup: MachineOwner
Can anyone point me in the right direction, in terms of making sense of this entry, and what I can do about it?
Answer
Just to make sure, I've performed some tests on my side, attaching to healthy processes and making dumps of just started process. In all cases output of !analyze -v was pretty similar to yours, except of the fact that my one is more verbose, I think it depends on debugger version.
For example, here is the output that I've got after attaching to just started Paint:
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/mspaint_exe/6_1_7600_16385/4a5bca29/ntdll_dll/6_1_7601_17725/4ec4aa8e/80000003/00050530.htm?Retriage=1
FAULTING_IP:
ntdll!DbgBreakPoint+0
00000000`76d90530 cc int 3
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000076d90530 (ntdll!DbgBreakPoint)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 0000000000000000
FAULTING_THREAD: 0000000000000cbc
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT
PROCESS_NAME: mspaint.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
EXCEPTION_PARAMETER1: 0000000000000000
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT
LAST_CONTROL_TRANSFER: from 0000000076e37ef8 to 0000000076d90530
STACK_TEXT:
FOLLOWUP_IP:
ntdll!DbgBreakPoint+0
00000000`76d90530 cc int 3
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!DbgBreakPoint+0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ec4aa8e
STACK_COMMAND: ~8s ; kb
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_80000003_ntdll.dll!DbgBreakPoint
BUCKET_ID: X64_APPLICATION_FAULT_STATUS_BREAKPOINT_ntdll!DbgBreakPoint+0
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/mspaint_exe/6_1_7600_16385/4a5bca29/ntdll_dll/6_1_7601_17725/4ec4aa8e/80000003/00050530.htm?Retriage=1
Followup: MachineOwner
---------
I've also taken a look at the explanation of ProcDump flags here: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx. It looks like using command line like
C:\apps\procdump.exe -accepteula -e -h -n 10 -t -w application.exe
you make procdump stop on every sign of hanging or exception without setting specific parameters, like memory usage number or procent of CPU usage.
I would suggest to use DebugDiag, which provides nice UI, where you could configure rules describing when dump should be created. Here is an explanation from me, how to gather a dump when you have excessive memory usage problem, or high CPU usage:
http://kate-butenko.blogspot.com/2012/06/how-to-gather-dump-with-debugdiag.html
and here is another fine screenshot-based explanation, how to get a dump in DebugDiag for specific exception:
From set of more lightweight tools you may also check the AdPlus tool (resides in C:\Program Files\Debugging Tools for Windows (x64) folder). I prefer DebugDiag, as it allows to catch specific types of exceptions.