Setting up WinDbg for .NET application crash analysis

There are many posts online on how to analyze a crash dump from a .NET process but I felt a need for a concise resource on how to get this done.  This blog post will take you from the point where you have a crash dump, typically a file with extension .hdmp to extracting information from the dump and determining the root cause of the crash.  For the purpose of this exercise, we will be assuming a dump file created off a 64 bit version of Windows Server 2008.

  • First you need to find an install the appropriate version of WinDbg.  Do a search on your system for WinDbg.  If you have Visual Studio 2013 or 2015, you should have both WinDbg (x86) and WinDbg (x64).  Select the latter.
  • Open WinDbg (x64) and configure the Symbol Search path as follows:
    1. Select File -> Symbol File Path
    2. In the resulting dialog, enter the following URL:  http://msdl.microsoft.com/download/symbols
    3. Select OK and return to the main application window.
  • Select File -> Open Crash Dump and locate the dump file ending in .hdmp.
  • Once the application has finished loading the symbols and extracted a minimal set of information from the dump file as follows:

Microsoft (R) Windows Debugger Version 6.3.9600.17237 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Temp\CrashDump\CrashDump\WERD999.tmp.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred http://msdl.microsoft.com/download/symbols
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Feb 1 15:20:11.000 2016 (UTC – 5:00)
System Uptime: 11 days 1:18:59.447
Process Uptime: 0 days 8:49:56.000
……………………………………………………….
…………………………………………
Loading unloaded module list
…..
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(6b8.f80): Unknown exception – code c0000374 (first/second chance not available)
ntdll!ZwWaitForSingleObject+0xa:
00000000`773cd9fa c3 ret

  • From the command WinDbg command line, type the following command:

!analyze -v

This will give you additional details on the crash, such as

ERROR: FindPlugIns 8007007b
ERROR: Some plugins may not be available [8007007b]
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
ntdll!RtlReportCriticalFailure+62
00000000`7743ffc2 eb00 jmp ntdll!RtlReportCriticalFailure+0x64 (00000000`7743ffc4)

EXCEPTION_RECORD: ffffffffffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 000000007743ffc2 (ntdll!RtlReportCriticalFailure+0x0000000000000062)
ExceptionCode: c0000374
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00000000774b7470

CONTEXT: 0000000000000000 — (.cxr 0x0;r)
rax=0000000019d00000 rbx=00000000000201cc rcx=0000000019d00000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=00000000773cd9fa rsp=000000001e92c358 rbp=ffffffffffffffff
r8=0000000000000000 r9=0000000000000040 r10=0000000000000000
r11=0000000000000286 r12=00000000774aa678 r13=0000000000b70000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll!ZwWaitForSingleObject+0xa:
00000000`773cd9fa c3 ret

DEFAULT_BUCKET_ID: WRONG_SYMBOLS

PROCESS_NAME: ServiceLauncher.exe

ERROR_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_PARAMETER1: 00000000774b7470

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

APP: servicelauncher.exe

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS

BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS

LAST_CONTROL_TRANSFER: from 0000000077440606 to 000000007743ffc2

STACK_TEXT:
00000000`1e92d660 00000000`77440606 : 00000000`00000002 000007fe`00000023 00000000`00000001 00000000`00000003 : ntdll!RtlReportCriticalFailure+0x62
00000000`1e92d730 00000000`77441812 : 00000000`00000065 000007fe`ea771f02 00000000`00000000 00000000`735d20da : ntdll!RtlpReportHeapFailure+0x26
00000000`1e92d760 00000000`774434f4 : 00000000`1d650000 00000000`1d650000 00000000`0000000a 00000000`735d20da : ntdll!RtlpHeapHandleError+0x12
00000000`1e92d790 00000000`774438d8 : 00000000`1d650000 00000000`1e3c13a0 00000000`00100000 00000000`00000001 : ntdll!RtlpLogHeapFailure+0xa4
00000000`1e92d7c0 00000000`773da5bf : 00000000`1d650000 00000000`1e3c13a0 00000000`1d650000 000007fe`ea771e70 : ntdll!RtlpAnalyzeHeapFailure+0x3a8
00000000`1e92d820 00000000`77181bba : 00000000`1d650000 00000000`00000001 00000000`1e3c13a0 00000000`1e3c13b0 : ntdll!RtlpFreeHeap+0x141f
00000000`1e92db60 00000000`73618d94 : 00000000`00000000 00000000`00000000 00000000`00000200 000007fe`ea7cc2ec : kernel32!HeapFree+0xa
00000000`1e92db90 00000001`80003329 : 00000000`00000080 00000001`80006350 00000000`04065072 00000000`1e92dc20 : msvcr100!free+0x1c
00000000`1e92dbc0 00000000`00000080 : 00000001`80006350 00000000`04065072 00000000`1e92dc20 00000000`00000001 : openSSLEncDec+0x3329
00000000`1e92dbc8 00000001`80006350 : 00000000`04065072 00000000`1e92dc20 00000000`00000001 00000000`1e518900 : 0x80
00000000`1e92dbd0 00000000`04065072 : 00000000`1e92dc20 00000000`00000001 00000000`1e518900 00000002`00000080 : openSSLEncDec+0x6350
00000000`1e92dbd8 00000000`1e92dc20 : 00000000`00000001 00000000`1e518900 00000002`00000080 00000000`1e92e0c0 : 0x4065072
00000000`1e92dbe0 00000000`00000001 : 00000000`1e518900 00000002`00000080 00000000`1e92e0c0 00000000`1a7f46d0 : 0x1e92dc20
00000000`1e92dbe8 00000000`1e518900 : 00000002`00000080 00000000`1e92e0c0 00000000`1a7f46d0 00000000`1e3c13b0 : 0x1
00000000`1e92dbf0 00000002`00000080 : 00000000`1e92e0c0 00000000`1a7f46d0 00000000`1e3c13b0 00000000`1e416c60 : 0x1e518900
00000000`1e92dbf8 00000000`1e92e0c0 : 00000000`1a7f46d0 00000000`1e3c13b0 00000000`1e416c60 00000000`1c48fa80 : 0x00000002`00000080
00000000`1e92dc00 00000000`1a7f46d0 : 00000000`1e3c13b0 00000000`1e416c60 00000000`1c48fa80 34303a72`6f727265 : 0x1e92e0c0
00000000`1e92dc08 00000000`1e3c13b0 : 00000000`1e416c60 00000000`1c48fa80 34303a72`6f727265 6c3a3237`30353630 : 0x1a7f46d0
00000000`1e92dc10 00000000`1e416c60 : 00000000`1c48fa80 34303a72`6f727265 6c3a3237`30353630 75663a29`34286269 : 0x1e3c13b0
00000000`1e92dc18 00000000`1c48fa80 : 34303a72`6f727265 6c3a3237`30353630 75663a29`34286269 3a293130`3128636e : 0x1e416c60
00000000`1e92dc20 34303a72`6f727265 : 6c3a3237`30353630 75663a29`34286269 3a293130`3128636e 31286e6f`73616572 : 0x1c48fa80
00000000`1e92dc28 6c3a3237`30353630 : 75663a29`34286269 3a293130`3128636e 31286e6f`73616572 00000000`00293431 : 0x34303a72`6f727265
00000000`1e92dc30 75663a29`34286269 : 3a293130`3128636e 31286e6f`73616572 00000000`00293431 00000000`00000000 : 0x6c3a3237`30353630
00000000`1e92dc38 3a293130`3128636e : 31286e6f`73616572 00000000`00293431 00000000`00000000 000007fe`ea77298c : 0x75663a29`34286269
00000000`1e92dc40 31286e6f`73616572 : 00000000`00293431 00000000`00000000 000007fe`ea77298c 00000000`1e3f83b0 : 0x3a293130`3128636e
00000000`1e92dc48 00000000`00293431 : 00000000`00000000 000007fe`ea77298c 00000000`1e3f83b0 000007fe`ea7924c5 : 0x31286e6f`73616572
00000000`1e92dc50 00000000`00000000 : 000007fe`ea77298c 00000000`1e3f83b0 000007fe`ea7924c5 00000000`1e3f83b0 : 0x293431
FOLLOWUP_IP:
openSSLEncDec+3329
00000001`80003329 ?? ???

SYMBOL_STACK_INDEX: 8

SYMBOL_NAME: opensslencdec+3329

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: openSSLEncDec

IMAGE_NAME: openSSLEncDec.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 566f3f6d

STACK_COMMAND: ~23s; .ecxr ; kb

FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000374_openSSLEncDec.dll!Unknown

BUCKET_ID: X64_APPLICATION_FAULT_WRONG_SYMBOLS_opensslencdec+3329

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:wrong_symbols_c0000374_opensslencdec.dll!unknown

FAILURE_ID_HASH: {e623d460-46a1-120d-b93f-282101a454d8}

Followup: MachineOwner

In the next post, we will analyze this crash dump in depth.

I also just learned today, that there are alternate commands that can be used to do a quick post modem on a crash dump such as: .loadby sos mscorwks and !clrstack.  This URL provides additional useful tips analyzing crash dumps via WinDbg.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s