Monthly Archives: March 2016

Setting up WinDbg for .NET application crash analysis

There are many posts online on how to analyze a crash dump from a .NET process but I felt a need for a concise resource on how to get this done.  This blog post will take you from the point where you have a crash dump, typically a file with extension .hdmp to extracting information from the dump and determining the root cause of the crash.  For the purpose of this exercise, we will be assuming a dump file created off a 64 bit version of Windows Server 2008.

  • First you need to find an install the appropriate version of WinDbg.  Do a search on your system for WinDbg.  If you have Visual Studio 2013 or 2015, you should have both WinDbg (x86) and WinDbg (x64).  Select the latter.
  • Open WinDbg (x64) and configure the Symbol Search path as follows:
    1. Select File -> Symbol File Path
    2. In the resulting dialog, enter the following URL:  http://msdl.microsoft.com/download/symbols
    3. Select OK and return to the main application window.
  • Select File -> Open Crash Dump and locate the dump file ending in .hdmp.
  • Once the application has finished loading the symbols and extracted a minimal set of information from the dump file as follows:

Microsoft (R) Windows Debugger Version 6.3.9600.17237 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Temp\CrashDump\CrashDump\WERD999.tmp.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred http://msdl.microsoft.com/download/symbols
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Feb 1 15:20:11.000 2016 (UTC – 5:00)
System Uptime: 11 days 1:18:59.447
Process Uptime: 0 days 8:49:56.000
……………………………………………………….
…………………………………………
Loading unloaded module list
…..
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(6b8.f80): Unknown exception – code c0000374 (first/second chance not available)
ntdll!ZwWaitForSingleObject+0xa:
00000000`773cd9fa c3 ret

  • From the command WinDbg command line, type the following command:

!analyze -v

This will give you additional details on the crash, such as

ERROR: FindPlugIns 8007007b
ERROR: Some plugins may not be available [8007007b]
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
ntdll!RtlReportCriticalFailure+62
00000000`7743ffc2 eb00 jmp ntdll!RtlReportCriticalFailure+0x64 (00000000`7743ffc4)

EXCEPTION_RECORD: ffffffffffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 000000007743ffc2 (ntdll!RtlReportCriticalFailure+0x0000000000000062)
ExceptionCode: c0000374
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00000000774b7470

CONTEXT: 0000000000000000 — (.cxr 0x0;r)
rax=0000000019d00000 rbx=00000000000201cc rcx=0000000019d00000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=00000000773cd9fa rsp=000000001e92c358 rbp=ffffffffffffffff
r8=0000000000000000 r9=0000000000000040 r10=0000000000000000
r11=0000000000000286 r12=00000000774aa678 r13=0000000000b70000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll!ZwWaitForSingleObject+0xa:
00000000`773cd9fa c3 ret

DEFAULT_BUCKET_ID: WRONG_SYMBOLS

PROCESS_NAME: ServiceLauncher.exe

ERROR_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_PARAMETER1: 00000000774b7470

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

APP: servicelauncher.exe

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS

BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS

LAST_CONTROL_TRANSFER: from 0000000077440606 to 000000007743ffc2

STACK_TEXT:
00000000`1e92d660 00000000`77440606 : 00000000`00000002 000007fe`00000023 00000000`00000001 00000000`00000003 : ntdll!RtlReportCriticalFailure+0x62
00000000`1e92d730 00000000`77441812 : 00000000`00000065 000007fe`ea771f02 00000000`00000000 00000000`735d20da : ntdll!RtlpReportHeapFailure+0x26
00000000`1e92d760 00000000`774434f4 : 00000000`1d650000 00000000`1d650000 00000000`0000000a 00000000`735d20da : ntdll!RtlpHeapHandleError+0x12
00000000`1e92d790 00000000`774438d8 : 00000000`1d650000 00000000`1e3c13a0 00000000`00100000 00000000`00000001 : ntdll!RtlpLogHeapFailure+0xa4
00000000`1e92d7c0 00000000`773da5bf : 00000000`1d650000 00000000`1e3c13a0 00000000`1d650000 000007fe`ea771e70 : ntdll!RtlpAnalyzeHeapFailure+0x3a8
00000000`1e92d820 00000000`77181bba : 00000000`1d650000 00000000`00000001 00000000`1e3c13a0 00000000`1e3c13b0 : ntdll!RtlpFreeHeap+0x141f
00000000`1e92db60 00000000`73618d94 : 00000000`00000000 00000000`00000000 00000000`00000200 000007fe`ea7cc2ec : kernel32!HeapFree+0xa
00000000`1e92db90 00000001`80003329 : 00000000`00000080 00000001`80006350 00000000`04065072 00000000`1e92dc20 : msvcr100!free+0x1c
00000000`1e92dbc0 00000000`00000080 : 00000001`80006350 00000000`04065072 00000000`1e92dc20 00000000`00000001 : openSSLEncDec+0x3329
00000000`1e92dbc8 00000001`80006350 : 00000000`04065072 00000000`1e92dc20 00000000`00000001 00000000`1e518900 : 0x80
00000000`1e92dbd0 00000000`04065072 : 00000000`1e92dc20 00000000`00000001 00000000`1e518900 00000002`00000080 : openSSLEncDec+0x6350
00000000`1e92dbd8 00000000`1e92dc20 : 00000000`00000001 00000000`1e518900 00000002`00000080 00000000`1e92e0c0 : 0x4065072
00000000`1e92dbe0 00000000`00000001 : 00000000`1e518900 00000002`00000080 00000000`1e92e0c0 00000000`1a7f46d0 : 0x1e92dc20
00000000`1e92dbe8 00000000`1e518900 : 00000002`00000080 00000000`1e92e0c0 00000000`1a7f46d0 00000000`1e3c13b0 : 0x1
00000000`1e92dbf0 00000002`00000080 : 00000000`1e92e0c0 00000000`1a7f46d0 00000000`1e3c13b0 00000000`1e416c60 : 0x1e518900
00000000`1e92dbf8 00000000`1e92e0c0 : 00000000`1a7f46d0 00000000`1e3c13b0 00000000`1e416c60 00000000`1c48fa80 : 0x00000002`00000080
00000000`1e92dc00 00000000`1a7f46d0 : 00000000`1e3c13b0 00000000`1e416c60 00000000`1c48fa80 34303a72`6f727265 : 0x1e92e0c0
00000000`1e92dc08 00000000`1e3c13b0 : 00000000`1e416c60 00000000`1c48fa80 34303a72`6f727265 6c3a3237`30353630 : 0x1a7f46d0
00000000`1e92dc10 00000000`1e416c60 : 00000000`1c48fa80 34303a72`6f727265 6c3a3237`30353630 75663a29`34286269 : 0x1e3c13b0
00000000`1e92dc18 00000000`1c48fa80 : 34303a72`6f727265 6c3a3237`30353630 75663a29`34286269 3a293130`3128636e : 0x1e416c60
00000000`1e92dc20 34303a72`6f727265 : 6c3a3237`30353630 75663a29`34286269 3a293130`3128636e 31286e6f`73616572 : 0x1c48fa80
00000000`1e92dc28 6c3a3237`30353630 : 75663a29`34286269 3a293130`3128636e 31286e6f`73616572 00000000`00293431 : 0x34303a72`6f727265
00000000`1e92dc30 75663a29`34286269 : 3a293130`3128636e 31286e6f`73616572 00000000`00293431 00000000`00000000 : 0x6c3a3237`30353630
00000000`1e92dc38 3a293130`3128636e : 31286e6f`73616572 00000000`00293431 00000000`00000000 000007fe`ea77298c : 0x75663a29`34286269
00000000`1e92dc40 31286e6f`73616572 : 00000000`00293431 00000000`00000000 000007fe`ea77298c 00000000`1e3f83b0 : 0x3a293130`3128636e
00000000`1e92dc48 00000000`00293431 : 00000000`00000000 000007fe`ea77298c 00000000`1e3f83b0 000007fe`ea7924c5 : 0x31286e6f`73616572
00000000`1e92dc50 00000000`00000000 : 000007fe`ea77298c 00000000`1e3f83b0 000007fe`ea7924c5 00000000`1e3f83b0 : 0x293431
FOLLOWUP_IP:
openSSLEncDec+3329
00000001`80003329 ?? ???

SYMBOL_STACK_INDEX: 8

SYMBOL_NAME: opensslencdec+3329

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: openSSLEncDec

IMAGE_NAME: openSSLEncDec.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 566f3f6d

STACK_COMMAND: ~23s; .ecxr ; kb

FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000374_openSSLEncDec.dll!Unknown

BUCKET_ID: X64_APPLICATION_FAULT_WRONG_SYMBOLS_opensslencdec+3329

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:wrong_symbols_c0000374_opensslencdec.dll!unknown

FAILURE_ID_HASH: {e623d460-46a1-120d-b93f-282101a454d8}

Followup: MachineOwner

In the next post, we will analyze this crash dump in depth.

I also just learned today, that there are alternate commands that can be used to do a quick post modem on a crash dump such as: .loadby sos mscorwks and !clrstack.  This URL provides additional useful tips analyzing crash dumps via WinDbg.

 

A cautionary tale on using C# destructors

It is well documented in the .NET community, that destructors, although available as a language construct, should be used with caution especially if its usage is to cleanup unmanaged resources.  The CLR’s garbage collector is responsible for releasing memory used by managed objects only.  It is, therefore, not aware of any native resources that may have been loaded by the application process, hence the need to explicitly dispose of these resources.

The object class, which is base or parent class of every object in .NET has a protected and virtual method named Finalize.   When a C# destructor is implemented in a class, this code is converted by the C# compiler to an overriding Finalize implementation.  During garbage collection, every instance of such an object which as an overridden Finalize implementation is added to an internal finalization queue. At some nondeterministic point, the garbage collector invokes the Finalize method of each object in the finalization queue giving it a chance to release its unmanaged resources.

For example, suppose you created a class called DirectoryReplicationService with the following constructs:


public class DirectoryReplicationService : IDirectoryReplicationService, IDisposable
{
[DllImport("DRSAPIHelper.dll", CallingConvention = CallingConvention.Winapi,
CharSet = CharSet.Unicode, EntryPoint = "DRSAPI_Bind", PreserveSig = true)]
public static extern Int32 DRSAPI_Bind(
string param1,
string param2,
out IntPtr phHelper);

[DllImport("DRSAPIHelper.dll", CallingConvention = CallingConvention.Winapi,
CharSet = CharSet.Unicode, EntryPoint = "DRSAPI_Unbind", PreserveSig = true)]
public static extern Int32 DRSAPI_Unbind(
IntPtr phHelper);

private readonly IntPtr _hHelper = IntPtr.Zero;

public DirectoryReplicationService(
String param1,
String param2,
)
{
int hRes = DRSAPI_Bind(param1, param2, out _hHelper);
if (!hRes.Equals(0))
{
Logger.Error("Failed to bind. " + hRes.ToString("X8"), this);
throw (new Exception("Failed to bind"));
}
}

~DirectoryReplicationService()
{
int hRes = DRSAPI_Unbind(_hHelper);

if (!hRes.Equals(0))
{
logger.Debug("Failed to unbind. " + hRes.ToString("X8"));
throw (new Exception("Failed to unbind"));
}

The line DRSAPI_Unbind in the destructor invokes the native resource _hHelper, which is responsible for releasing its own resources.  When compiled, the destuctor code is replaced by


protected override Finalize()
{
int hRes = DRSAPI_Unbind(_hHelper);

if (!hRes.Equals(0))
{
logger.Debug("Failed to unbind. " + hRes.ToString("X8"));
throw (new Exception("Failed to unbind"));
}

There are fundamentally two problems here, which will be manifested as a random crash in the application:

  1. Relying on the Finalize method to release native resources, which is called in a non-deterministic fashion, could cause a memory violation, especially if there are background threads in the application that could potentially create additional instances of the DirectoryReplicationService.
  2. Throwing an exception in a destructor is not recommended as this could effectively cause an application crash.  MSDN states:

If Finalize or an override of Finalize throws an exception, and the runtime is not hosted by an application that overrides the default policy, the runtime terminates the process and no activetry/finally blocks or finalizers are executed. This behavior ensures process integrity if the finalizer cannot free or destroy resources.

So, what do we do?

As recommended, the best way to rid of native resources in a managed application is with proper implementation of the Dispose pattern.  Replacing the desctuctor with the following code:


public class DirectoryReplicationService

{

public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}

private void Dispose(bool disposing)
{
if (disposing)
{
int hRes = DRSAPI_Unbind(_hHelper);

if (!hRes.Equals(0))
{
Logger.Error("Failed to unbind. " + hRes.ToString("X8"), this);
}
}
}

}

ensures that any native resources are properly released in a deterministic fashion, thereby preventing opportunities for memory leaks and random crashes due to memory access violations.

This is what we observed in one of our production ready applications.