Category Archives: .NET

c# volatile demystified -my take

According to MSDN (http://msdn.microsoft.com/en-us/library/6kac2kdh.aspx)
a thread is the basic operating system entity that is used to execute a given task defined by a set of instructions. Therefore, a thread has its context, which includes all information it needs to resume execution such as set of CPU registers and stack of local variables, all in the address space of the thread’s host process.

Consider the following class:


public EncryptionService
{
public static readonly EncryptionService Instance = new EncryptionService();
private EncryptionService(){}
private int _counter = 0;
public int Encrypt(string input)
{
// run some encryption algorithm
return ++counter;
}
}

This class is singleton because it contains a private constructor and you can only access the single Instance. Since there is only a single instance of EncryptionService, there is a chance
that multiple threads could be running Encrypt and updating the local variable _counter.  In other words, the class is not thread safe.

Now, recall the thread context we talked about.  When a thread is created, the CLR allocates it 1MB stack, atleast for x86 CPU architectures.

The stack space is used for passing arguments to a method and for holding local variables defined in the method. Before executing a method the CLR executes prologue code [REF CLR via C#] that initializes the method, such as getting its execution and return address and allocating memory for its local variables on the thread’s stack, which is part of the thread’s execution context. When the thread executes more methods, its stack fills up with local variables from these methods until the method’s epilogue code runs clearing all of the variables that should not [or be] out of scope.

While the unit of execution is a thread, the physical entity which executes code, that have been translated to machine instructions is the CPU.  Naturally, this means that those variables defined on the thread’s stack are stored in the CPU cache.  When a thread is executing code on a specific CPU, its variables are therefore stored in this particular CPU’s cache.

So what exactly happens when multiple threads execute a method on a single instance of our EncryptionService?  We first need to understand how the above C# program is converted to assembly instructions which is what the CPU executes.

The Microsoft C# compiler generates Microsoft Intermediate Language (MSIL).  Which is what is stored in the assembly.  When the CLR loads this assembly and attempts to invoke the Decrypt method, it Just-in-Time (JIT) compilers the code to CPU specific assembly language instructions, storing this in a location in memory, at which point execution transfers to the memory location where the Decrpyt method is saved.

When the executing thread invokes Decrypt, is encounters a local variable called _counter stored at a certain memory location.  This variable has to be loaded from its memory location into the CPU’s register.  If this memory location was not already in any of the CPU’s caches, a cache miss occurs and this data and sorrounding bytes, as defined by the cache line size, is retrieved from memory and saved in the CPU cache, then loaded into the CPU register.  Then the CPU updates the register value, as part of the add instruction, and then stores the changed value in the respective cache. For this new value to become visible to other processes, or threads, it needs to be written back to main memory.  When exactly this happens is defined by what is known as a CPU write policy.

If you have two threads doing the same thing, it could happen that the values of _counter in each CPU’s cache is not consistent, since each CPU was working on its local local copy of the _counter variable in its registers/cache. Based on your use case, this could not be a desirable outcome.

More on Static Variables

When you decorate a variable as static, interesting things start to happen.

A static type is defined as part of the type definition. What this effectively means is that a static field is defined in the the header block of the type definition object.

This means that the bytes backing these static fields are part of the type’s header block. And since there is just one type header block, we have only one static field in the entire application domain.

Since a static field is backed by a memory block defined as part of the object’s type header which is at a fixed location on the managed heap, several threads can potentially override the work of each other.

This is a similar effect to having a s single instance object being operated upon my multiple threads without any synchronization mechanism.

On Chip Cache Memory
As noted before, a static member is defined at a fixed location in the object’s type header block in memory.  This means that instructions executing on the CPU need to fetch this static variable in memory to write or read from it.

To improve performance, today’s CPUs have something called a Cache Memory, which is a set of CPU registers defined on the CPU chip [ref??].  These are called Level caches, based on how close they are to the CPU.  There is an L1 cache, L2 and L3 cache, with the L1 cache resident on the CPU chip. “L” standing for level.   This means that instead of accessing RAM via the memory bus [TODO PICS], the CPU simply refers to its register cache to retrieve a value of a variable.

The first time a thread needs to read a value from memory such as our static variable, the CPU will read this value and various surrounding bytes, also known as the cache-line and store this cache line it in its on-chip cache.  You can imagine an on-chip cache as simply a dictionary of of key/value pairs of cache lines, keyed on memory address or address range.  These address keys are actually known as tag and each value of this key-value pair is a cache entry that comprises of actual data and some sort of flag indicating if this field has been updated from the last time it was read from memory.

 

When the value needs to be updated, the CPU updates its registry cache instead of writing directly to static field in memory. Periodically, the CPU then flushes out all of its content to memory (RAM).

Static Variables Concurrency
With all of this in mind, let us examine what happens with our static variable, within a multi-threaded, multi-core execution context.\


public EncryptionService

{

public static readonly EncryptionService Instance = new EncryptionService();
private EncryptionService(){}
private int _counter = 0;
public int Encrypt(string input)
{
// run some encryption algorithm
return ++counter;
}

public int Decrypt(string input)
{
// run some encryption algorithm
return ++counter;
}
}

Say thread A running on CPU A is about to execute Encrypt so it reads bytes in memory
for _counter variable. This variable is now in CPU A’s cache as shown below:

Now a second thread, thread B running on CPU B, attempts to execute Encrypt and
copys the value of _counter from memory, just before CPU A has had a chance to
flush out its update to the memory, recall this is a singleton.
The content of CPU B’s cache is now different from what is in memory for the value of _counter.

This is a classic cache concurrency problem and happens because CPU flushes happen at an unpredictable time in the future. Although some CPUs provide an ability to control this problem such as the IA64 with volatile read with write acquire semantics.

Volatile Read/Write
Recall a cache concurrency happens when one CPU updates a value in its on chip cache and before it has flushed this change to memory, another CPU has read the previous value from memory.  So the new flushed value is not reflected in the second CPU’s cache.

Volatile write means that a CPU writes a value of a register out to its cache and then flushes the entire content of the CPU cache to memory.   Interesting choice of word volatile.

Volatile read means that a CPU reads a value into memory and invalidates its cache?  Invalidating the cache means that this register value is no longer current and therefore, the next subsequent read of this value will come from memory.  Hence the word volatile, which means liable to change rapidly or unpredictably.  The fact that CPU register read and write is causing rapid flushing and invalidation of the CPU cache is reason why these are referred to as volatile operations, since CPU caches are meant to do otherwise.

There is also a Memory Fence which flushes the cache content to memory and then invalidates the cache.

How does this prevent cache concurrency?


public EncryptionService
{
public static readonly EncryptionService Instance = new EncryptionService();
private EncryptionService(){}
private int _counter = 0;
public int Encrypt(string input)
{
var current = Thread.VolatileRead(ref _counter); //1
Thread.VolatileWrite(ref _counter, current + 1); //2
return _counter;
}

public int Decrypt(string input)
{
// run some decryption algorithm
var current = _counter;
counter = current + 1;
return _counter;
}
}

With these changes, this is what happens now.

When thread A on CPU A executes line 1, it performs a volatile read meaning that is reads the current value of _counter from memory but invalidates it cache, so that the next subsequent read of this value comes from memory.  If thread B now modifies that same memory in its CPU register location, since the field is marked a volatile, CPU B will also flush out the cache line containing this variable to memory.

In C#, this is what methods VolatileRead and VolatileWrite do.

c# volatile To The Rescue

All of the above is nice but cumbersome to use so the C# team provided a simple keyword volatile that simply wraps reading and writing to a variable in volatile read/write.


public EncryptionService
{
public static readonly EncryptionService Instance = new EncryptionService();
private EncryptionService(){}
private volatile int _counter = 0;
public int Encrypt(string input)
{
var current = _counter; //1
_counter, current + 1; //2
return _counter;
}

public int Decrypt(string input)
{
// run some decryption algorithm
var current = _counter;
counter = current + 1;
return _counter;
}
}

This means that a CPU does not cache this variable and reads/writes directly to memory. So while this solves cache concurrency, there is still a problem since multiple threads can attempt to write to this memory location at the same time.

To prevent this you have to use C# thread synchronization constructs,

Note that this problem, while not the same as two threads writing to/reading from the same memory location, manifest the same way as an application problem. This is a problem resulting from running on a multi-core machine.

 

References:

http://computer.howstuffworks.com/computer-memory3.htm

http://arstechnica.com/gadgets/2002/07/caching/2/

http://en.wikipedia.org/wiki/CPU_cache

https://www.amazon.com/CLR-via-Second-Pro-Developer/dp/B002V1H004

 

Sample Quiz on Delayed Execution of IEnumerable

A sample quiz to test your knowledge of delayed execution of IEnumerable<T>.

Assume you have a method as follows:


public IEnumerable<AppointmentSlot> GetOpenSlots(Dictionary<string, string> request)
{
return _openSlots;
}

The above method returns a list of open slots which could simply have been a collection prepoluated when the class was constructed.

Say somewhere else in the code you call this method as follows:


public RelayCommand GetOpenAppointmentSlotsCmd
{
get { return new RelayCommand(o => true, o =>
{
OpenAppointmentSlots = _resourcesServiceClient
.GetOpenSlots(null)
.Select(s => new AppointmentSlotViewModel(s));
});}
}

Where OpenAppointmentSlots is a property of type IEnumerable<T> and T is of type AppointmentSlotViewModel which in simple form could look something like  this:


public class AppointmentSlotViewModel : ViewModelBase
{
public AppointmentSlot AppointmentSlot { get; }

public AppointmentSlotViewModel(AppointmentSlot slot)
{
AppointmentSlot = slot;
}

public DateTime StartsAt => AppointmentSlot.StartsAt;
public int DurationInMiutes => AppointmentSlot.DurationInMiutes;
public string ProviderId => AppointmentSlot.ProviderId;

public RelayCommand BookAppointmentSlot
{
get
{
return new RelayCommand(o => NotBooked, o =>
{
NotBooked = false;
});
}
}

private bool _notBooked = true;
public bool NotBooked
{
get { return _notBooked; }
set
{
SetField(ref _notBooked, value, "NotBooked");
}
}
}

Further assume you had the list of OpenAppointmentSlots bound to a ListView and had a DataTemplate which manipulated the NotBooked property of AppointmentSlotViewModel.

Now, say somewhere else in the code, perhaps in the same view model which gets the list of open slots, you attempt to submit these selected open slots for booking as follows:


public RelayCommand BookAppointmentCmd
{
get
{
return new RelayCommand(o => true, o =>
{
var booked = OpenAppointmentSlots..Where(s => s.IsBooked()).Select(nb => nb.AppointmentSlot);
BookedAppointmentStatus = _resourcesServiceClient.BookAppointments(booked);
});
}
}

After manipulating the UI such that you now have two slots booked,  what do you suppose will be the result of the variable called booked, in the above snippet and why?

 

Yet another practical pocket guide on writing clean code

Being a big proponent on crafting beautiful, robust and maintainable code, I have read several books and articles on the subject.  One of my favorite resources is a book titled “Clean Code” by Robert C. Martin.

I would propose that such a resource be readily available in the company library and would even go as far as advocating for every software engineer joining the company to either have read this book or is required to read this book as part of the on-boarding process.

In summary what is clean code:  Code which mostly abides by the SOLID principles of software design.  In my own words.

Methods should have a single responsibility.  It is better to have a class with many small methods that have a class with a small number of large methods.

Methods should not be overly long. A well known and acceptable measure is that a method should not span the entire code editor space in Visual Studio, when viewed over a 15″ laptop monitor.  For example, this is a long method:

On the otherhand, this is a nice, short and terse method:

Methods should return early if possible.  This avoid too many nested iffs.  For example, consider the following method which takes in input object of some sort

   private void InitializeActionMethods()
        {
            if (_configurationManager.Configuration == null)
            {
                Logger.Warn("Some configuration is not defined.");
                return;
            }

We fail fast and early.  In contrast, we could written the code like this:

   private void InitializeActionMethods()
        {
            if (_configurationManager.Configuration =! null)
            {
                // continue
            }

This creates a code base with too many nested-iffs which is hard to read and maintain.

Method names and variables should clearly indicate purpose.  I often say code is a story.  Write code as if you are writing a story.  Books with shorter paragraphs are more engaging than books with longer paragraphs.  I often see developers naming variables using acronyms instead of taking the time to craft out descriptive variable names. Again, if code is a story, we need to clearly identify the characters.

Entities themselves should have single responsibility.  This one is also easy to violate.  I have seem some very large and weird looking classes over the years.  I have also seen classes that are almost impossible to refactor and unit test as it is composed of a collection of large, deeply nested methods with a large number of inter-dependencies.  Keep classes small.  I have told my devs that is is better to have a code base with thousands of small entities that one with a small number of large entities.  The former system, if well organized, is easily to reason with, maintain, modular and robust.

Entities should have dependencies passed to them.  The term coined for this is dependency injection or inversion of control.  I always get confused here but the idea is for a factory to construct a car, it needs to have all of its dependent bits, such as assembly line, etc.  These must be explicit and defined up front.

Unit test, unit tests and more unit tests. I cannot emphasize this enough but any component in the system should have an associated unit test which is concise.  There are well documented strategies for crafting awesome unit test but they should abide by the AAA principle of Arrange, Act and Assert.  Google this. Also, make these test very easy to follow.  All dependencies should be arranged or created up front.  if you are resolving entities from some container somewhere, which includes configuring some sort of logger, which requires some additional piece of configuration somewhere, you probably need to step back and rethink your tests and class design.

Code should be closed for modification and open for extension, as stated in the Open/Closed Principle, code should be easy to extend but closed for modification.  This is a tough one but think if it this way.  If you start creating code with long switches, then it is time to sit back and think of some patterns to use.

Code should be robust against anomalies but at the same time need not be overly micro-optimized.  Beautiful code means that it is easy on the eye, easy on the mind, free flowing, yet robust against extremities.  This includes excellent exception handling and logging.

Happy Coding.

Unable to start debugging on a web server. Could not start ASP.NET debugging.

If you attempt to debug an ASP.NET web application and get the following error, you can Google it and find a lot of hits.  There are also numerous posts on StackOverflow for this error but interestingly none of the answers in these posts helped for me.

What worked for me was simple.

The application was being deployed in the DefaultApp pool with the following properties:

  • .NET CLR version = 4.0
  • Managed Pipeline = Integrated
  • Identity = domain/my-user-name

I changed the target application pool to one called .NET 4.5 with the following properties:

  • .NET CLR version = 4.0
  • Managed Pipeline = Integrated
  • Identity = ApplicationPoolIdentity

With this change, I was able to run the application in debug mode.

Happy coding.

 

 

Be on lookout for StackExchange Redis ConnectionMultiplexer ConnectionFailed event misfires

We are using the StackExchange.Redis ConnectionMultiplexer class to manage our connections to Redis.  Without clear guidance from the documentation, we have attempted to create our own retry strategy with this client with code which looks like this:

 

  public RedisClient()
 : this(GetConnectionString())
 {
 }
 
 public RedisClient(string connectionString)
 {
 _logger = LogServiceProvider.Instance.GetLogger(GetType());
 _connectionString = connectionString;
 MaxRetryAttempts = 10;
 DelayBeforeRetryInMillisecs = 500;
 InitializeConnection();
 }

 private void InitializeConnection()
 {
 _logger.Info("Initializing a connection to the Redis cluster. ");
 bool isReconnectionAttempt = false;

 if (_connectionMultiplexer != null)
 {
 Debug.WriteLine("disposing " + _connectionMultiplexer.GetHashCode());
 _connectionMultiplexer.ConnectionFailed -= HandleConnectionFailedEvent;

 // test this change.....
 _connectionMultiplexer.Close(false);
 isReconnectionAttempt = true;
 _logger.Info("This is reconnection attempt to the Redis cluster.");
 }

 _connectionMultiplexer = ConnectionMultiplexer.Connect(_connectionString);
 _needConnect = !_connectionMultiplexer.IsConnected;
 _connectionMultiplexer.ConnectionFailed += HandleConnectionFailedEvent;

 Debug.WriteLine("created " + _connectionMultiplexer.GetHashCode());

 if (!_needConnect && isReconnectionAttempt)
 {
 _logger.Info("Reconnection to the Redis cluster was succeeded.");
 RaiseRedisConnectionReestablished();
 }

 if (!_needConnect)
 {
 _logger.Info("Connection is successfully established to the Redis cluster.");
 }
 else
 {
 _logger.Error("Cannot establish a connection to the Redis cluster. ");
 }
 }

 private void HandleConnectionFailedEvent(object sender, ConnectionFailedEventArgs args)
 {
 // we could potentially receive multiple of these events so we need to be be careful
 Debug.WriteLine( "received connection failure event from " + sender.GetHashCode());
 Debug.WriteLine(" multiplexer id is " + _connectionMultiplexer.GetHashCode());

 // There's a known issue with the Redis ConnectionMultiplexer which prevents if from 
 // completely releasing an event handler even after it has been disconnected and disposed.
 // Se we need the following line of code.

 if (sender.GetHashCode() != _connectionMultiplexer.GetHashCode())
 return;

 _logger.Error("Connection to the Redis cluster has failed with the following event arg:", args.Exception);

 RaiseRedisConnectionFailed(args.Exception);
 AttemptReconnection(1);
 }

 private void AttemptReconnection(int trial = 1)
 {
 if (trial == MaxRetryAttempts)
 {
 _logger.Info("Have attempted to re-connect to the Redis cluster 3 times. Aborting.");
 
 return;
 }

 if (_connectionMultiplexer.IsConnected)
 {
 _logger.Info("Connetion to Redis cluster is no longer required.");
 return;
 }

 _logger.InfoFormat("Attempting reconnection attempt {0} to the Redis cluster.", trial);
 RedisReconnectionAttempt(this, new RedisReconnectionAttemptEventArgs(trial));

 // wait for a few seconds and then try to reconnect.....
 Thread.Sleep(DelayBeforeRetryInMillisecs);
 InitializeConnection();

 trial = trial + 1;
 TotalNumberOfReconnectionAttempts = TotalNumberOfReconnectionAttempts + 1;
 AttemptReconnection(trial);
 
 }

When a network drop is simulated, the ConnectionFailed event is fired as expected. When this happens, an attempt is made to dispose of the current instance of the ConnectionMultiplexer object and create a new one. We do this to avoid the situation where attempting to access the current ConnectoinMultiplexer instance throws an exception indicating the object has already been disposed.

So we dispose the current instance, or atleast we think we do and create a new one. Yet somehow, the original instance which experienced the network drop, but is now disposed, still manages to fire ConnectionFailed events even though we are no longer supposed to be listerning.  After all, we unsubscribed to the event as indicated in the InitializeConnection method above.

I could not get an answer from StackOverflow. I also could not determine the cause just by looking at the source code for this ConnectionMultiplexer class.  What I could do, however, is put in a small hack to ensure these multiple noisy events are ignored, unless they are coming from the correct ConnectionMultiplexer instance.

I even have test to ensure my hack works.

Happy coding.

WPF Grid Horizontal andVertical Seperator

This gets me every time… so I will blog this and will never have to Google around for it.  Hopefully, when I do so, this blog post appears top in the Google search results.

I need a WPF Grid Separator, basically a visual line that allows two parts of a page to be resized…..

I google for “WPF horizontal seperator” and the closest thing that came around was this blog post .  Close but not quite, so I though I could just translate the Width=”5″ to Heigh=”5″ and HorizontalAlignment=“Stretch” to VerticalAlignment=”Stretch”. Those translations do not quite work. So here it is in one place, the XAML for a WPF Horizontal and Vertical grid splitter.

At this point, my XAML looked like this:


<Grid>
<Grid.RowDefinitions>
<RowDefinition Height="*"/>
<RowDefinition Height="5"/>
<RowDefinition Height="*"/>
</Grid.RowDefinitions>

<Grid.ColumnDefinitions>
<ColumnDefinition Width="*"/>
<ColumnDefinition Width="5"/>
<ColumnDefinition Width="*"/>
</Grid.ColumnDefinitions>
<GridSplitter Grid.Column="1" Width="5" HorizontalAlignment="Stretch" />
<GridSplitter Grid.Row="1" Height="5" HorizontalAlignment="Stretch" />
</Grid>

Happy coding…

Setting up WinDbg for .NET application crash analysis

There are many posts online on how to analyze a crash dump from a .NET process but I felt a need for a concise resource on how to get this done.  This blog post will take you from the point where you have a crash dump, typically a file with extension .hdmp to extracting information from the dump and determining the root cause of the crash.  For the purpose of this exercise, we will be assuming a dump file created off a 64 bit version of Windows Server 2008.

  • First you need to find an install the appropriate version of WinDbg.  Do a search on your system for WinDbg.  If you have Visual Studio 2013 or 2015, you should have both WinDbg (x86) and WinDbg (x64).  Select the latter.
  • Open WinDbg (x64) and configure the Symbol Search path as follows:
    1. Select File -> Symbol File Path
    2. In the resulting dialog, enter the following URL:  http://msdl.microsoft.com/download/symbols
    3. Select OK and return to the main application window.
  • Select File -> Open Crash Dump and locate the dump file ending in .hdmp.
  • Once the application has finished loading the symbols and extracted a minimal set of information from the dump file as follows:

Microsoft (R) Windows Debugger Version 6.3.9600.17237 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Temp\CrashDump\CrashDump\WERD999.tmp.hdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred http://msdl.microsoft.com/download/symbols
Symbol search path is: http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon Feb 1 15:20:11.000 2016 (UTC – 5:00)
System Uptime: 11 days 1:18:59.447
Process Uptime: 0 days 8:49:56.000
……………………………………………………….
…………………………………………
Loading unloaded module list
…..
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(6b8.f80): Unknown exception – code c0000374 (first/second chance not available)
ntdll!ZwWaitForSingleObject+0xa:
00000000`773cd9fa c3 ret

  • From the command WinDbg command line, type the following command:

!analyze -v

This will give you additional details on the crash, such as

ERROR: FindPlugIns 8007007b
ERROR: Some plugins may not be available [8007007b]
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
ntdll!RtlReportCriticalFailure+62
00000000`7743ffc2 eb00 jmp ntdll!RtlReportCriticalFailure+0x64 (00000000`7743ffc4)

EXCEPTION_RECORD: ffffffffffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 000000007743ffc2 (ntdll!RtlReportCriticalFailure+0x0000000000000062)
ExceptionCode: c0000374
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 00000000774b7470

CONTEXT: 0000000000000000 — (.cxr 0x0;r)
rax=0000000019d00000 rbx=00000000000201cc rcx=0000000019d00000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=00000000773cd9fa rsp=000000001e92c358 rbp=ffffffffffffffff
r8=0000000000000000 r9=0000000000000040 r10=0000000000000000
r11=0000000000000286 r12=00000000774aa678 r13=0000000000b70000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll!ZwWaitForSingleObject+0xa:
00000000`773cd9fa c3 ret

DEFAULT_BUCKET_ID: WRONG_SYMBOLS

PROCESS_NAME: ServiceLauncher.exe

ERROR_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_PARAMETER1: 00000000774b7470

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

APP: servicelauncher.exe

ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre

MANAGED_STACK: !dumpstack -EE
No export dumpstack found

PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS

BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS

LAST_CONTROL_TRANSFER: from 0000000077440606 to 000000007743ffc2

STACK_TEXT:
00000000`1e92d660 00000000`77440606 : 00000000`00000002 000007fe`00000023 00000000`00000001 00000000`00000003 : ntdll!RtlReportCriticalFailure+0x62
00000000`1e92d730 00000000`77441812 : 00000000`00000065 000007fe`ea771f02 00000000`00000000 00000000`735d20da : ntdll!RtlpReportHeapFailure+0x26
00000000`1e92d760 00000000`774434f4 : 00000000`1d650000 00000000`1d650000 00000000`0000000a 00000000`735d20da : ntdll!RtlpHeapHandleError+0x12
00000000`1e92d790 00000000`774438d8 : 00000000`1d650000 00000000`1e3c13a0 00000000`00100000 00000000`00000001 : ntdll!RtlpLogHeapFailure+0xa4
00000000`1e92d7c0 00000000`773da5bf : 00000000`1d650000 00000000`1e3c13a0 00000000`1d650000 000007fe`ea771e70 : ntdll!RtlpAnalyzeHeapFailure+0x3a8
00000000`1e92d820 00000000`77181bba : 00000000`1d650000 00000000`00000001 00000000`1e3c13a0 00000000`1e3c13b0 : ntdll!RtlpFreeHeap+0x141f
00000000`1e92db60 00000000`73618d94 : 00000000`00000000 00000000`00000000 00000000`00000200 000007fe`ea7cc2ec : kernel32!HeapFree+0xa
00000000`1e92db90 00000001`80003329 : 00000000`00000080 00000001`80006350 00000000`04065072 00000000`1e92dc20 : msvcr100!free+0x1c
00000000`1e92dbc0 00000000`00000080 : 00000001`80006350 00000000`04065072 00000000`1e92dc20 00000000`00000001 : openSSLEncDec+0x3329
00000000`1e92dbc8 00000001`80006350 : 00000000`04065072 00000000`1e92dc20 00000000`00000001 00000000`1e518900 : 0x80
00000000`1e92dbd0 00000000`04065072 : 00000000`1e92dc20 00000000`00000001 00000000`1e518900 00000002`00000080 : openSSLEncDec+0x6350
00000000`1e92dbd8 00000000`1e92dc20 : 00000000`00000001 00000000`1e518900 00000002`00000080 00000000`1e92e0c0 : 0x4065072
00000000`1e92dbe0 00000000`00000001 : 00000000`1e518900 00000002`00000080 00000000`1e92e0c0 00000000`1a7f46d0 : 0x1e92dc20
00000000`1e92dbe8 00000000`1e518900 : 00000002`00000080 00000000`1e92e0c0 00000000`1a7f46d0 00000000`1e3c13b0 : 0x1
00000000`1e92dbf0 00000002`00000080 : 00000000`1e92e0c0 00000000`1a7f46d0 00000000`1e3c13b0 00000000`1e416c60 : 0x1e518900
00000000`1e92dbf8 00000000`1e92e0c0 : 00000000`1a7f46d0 00000000`1e3c13b0 00000000`1e416c60 00000000`1c48fa80 : 0x00000002`00000080
00000000`1e92dc00 00000000`1a7f46d0 : 00000000`1e3c13b0 00000000`1e416c60 00000000`1c48fa80 34303a72`6f727265 : 0x1e92e0c0
00000000`1e92dc08 00000000`1e3c13b0 : 00000000`1e416c60 00000000`1c48fa80 34303a72`6f727265 6c3a3237`30353630 : 0x1a7f46d0
00000000`1e92dc10 00000000`1e416c60 : 00000000`1c48fa80 34303a72`6f727265 6c3a3237`30353630 75663a29`34286269 : 0x1e3c13b0
00000000`1e92dc18 00000000`1c48fa80 : 34303a72`6f727265 6c3a3237`30353630 75663a29`34286269 3a293130`3128636e : 0x1e416c60
00000000`1e92dc20 34303a72`6f727265 : 6c3a3237`30353630 75663a29`34286269 3a293130`3128636e 31286e6f`73616572 : 0x1c48fa80
00000000`1e92dc28 6c3a3237`30353630 : 75663a29`34286269 3a293130`3128636e 31286e6f`73616572 00000000`00293431 : 0x34303a72`6f727265
00000000`1e92dc30 75663a29`34286269 : 3a293130`3128636e 31286e6f`73616572 00000000`00293431 00000000`00000000 : 0x6c3a3237`30353630
00000000`1e92dc38 3a293130`3128636e : 31286e6f`73616572 00000000`00293431 00000000`00000000 000007fe`ea77298c : 0x75663a29`34286269
00000000`1e92dc40 31286e6f`73616572 : 00000000`00293431 00000000`00000000 000007fe`ea77298c 00000000`1e3f83b0 : 0x3a293130`3128636e
00000000`1e92dc48 00000000`00293431 : 00000000`00000000 000007fe`ea77298c 00000000`1e3f83b0 000007fe`ea7924c5 : 0x31286e6f`73616572
00000000`1e92dc50 00000000`00000000 : 000007fe`ea77298c 00000000`1e3f83b0 000007fe`ea7924c5 00000000`1e3f83b0 : 0x293431
FOLLOWUP_IP:
openSSLEncDec+3329
00000001`80003329 ?? ???

SYMBOL_STACK_INDEX: 8

SYMBOL_NAME: opensslencdec+3329

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: openSSLEncDec

IMAGE_NAME: openSSLEncDec.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 566f3f6d

STACK_COMMAND: ~23s; .ecxr ; kb

FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000374_openSSLEncDec.dll!Unknown

BUCKET_ID: X64_APPLICATION_FAULT_WRONG_SYMBOLS_opensslencdec+3329

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:wrong_symbols_c0000374_opensslencdec.dll!unknown

FAILURE_ID_HASH: {e623d460-46a1-120d-b93f-282101a454d8}

Followup: MachineOwner

In the next post, we will analyze this crash dump in depth.

I also just learned today, that there are alternate commands that can be used to do a quick post modem on a crash dump such as: .loadby sos mscorwks and !clrstack.  This URL provides additional useful tips analyzing crash dumps via WinDbg.