Interpreting your agile team’s velocity from a mathematical perspective

This has always been a contentious topic and one which has always been up for debate in all the companies I have worked for.

From my understanding, a sprint velocity is supposed to be an approximate, and not an absolute measure, on what a team can accomplish within a given sprint taking into consideration certain assumption and environmental factors.   This number, from my understanding, is simply a guideline but should not be used to solely determine what a team can accomplish in future sprints.  Some teams view this number as sheer complexity of the task.  Others view this number as a combination of complexity and effort.  The beauty of agile is each team decides how exactly they want to interpret this number.   That’s an internal implementation of the team.  For the sake of this exercise, we will view this number as relative complexity of a task at hand, which somewhat is related to effort assuming no external factors.

For our Rabbit Hat Mechanics agile team,  let’s model a sprint as follows:

y = ƒ(c_1x_1,c_2x_2,c_3x_3, c_4x_4)   where

  • x_1 = previous points from last sprint
  • x_2 = number of staff on holidays or sick
  • x_3 = team familiarity with tooling and working environment
  • x_4 = environmental stability

and c_1 to c_4 and constants.

I have modeled a sprint using four possibly independent variables but each team is different and some could possibly be modeled using more than four variables. Some could be

  • x_5 = capacity of team leads
  • x_6 = quality of scrum master/manager
  • x_7 = number of company meetings
  • x_8 = quality of requirements

This is multi-variable equation with several input parameters one of which is the previous sprint velocity.  The point here is, since human resource management is never an absolute science,  is more of a nonlinear and possibly nth order differential equation, one can appreciate that the previous sprint velocity is just one factor, albeit a heavy one, which should go into the equation while determining what the team can deliver.

When folks advocate for the fact that the previous sprint velocity of, say x_1, should be the only variable or determinant to be considered when determining the next sprint throughput, what they are effectively saying is that the above equation reduces to a linear one in the form

y = ƒ (x_1)

and from my experience dealing with people and teams, this is simply not the case. Humans are not robots and even with robots, there is usually a discrepancy between theory and reality and that x_1 = 40 points from the last sprint will not necessarily be the same number of points a robot can accomplish this sprint taking into consideration external factors.

So, I am saying treat the sprint velocity for what it is:

A parameter, and not the only parameter, which you should take into consideration when determining the team’s potential through put for next sprint.


10 Golden Rules for Unit Tests

There are some rules of thumb I encourage my team to follow when writing unit tests and they are as follows:

1. System under test should be clearly defined.  A variable naming convention I have personally adopted is “sut” to stand for “system under test”.

2.  Each test should abide by the AAA principle of Arrange, Act and Assert.  There should be a clear demarcation between these three different aspects of a test.

3.  As part of the Arrange phase, all dependencies of the system under test should be clearly defined, and mocks used where to abstract away external dependencies, whenever possible. Also, always best to mock via interfaces instead of abstract base classes. No container registrations here and no hidden dependency injection into the system under test.

4.  As part of the Arrange phase, all additional dependencies such as file names or configuration items should be explicit and clearly defined. Avoid depending on external entities which will not guarantee consistency in your test results such as webservices, etc.

5.  During the Acting phase, it should be clear which of the APIs of the system under test is being exercised and this method being exercised should typically return a response.

6.  Assertion should be based upon the response of the method acted upon from the previous phase.  Assert the response is what you expect.  Avoid asserting that the method under test was called.  For example, this is a good assertion:

// arrange
var sut = new ResourceController();

// act
var result = sut.GetResource(new Dictionary()).ToList();
// assert
Assert.AreEqual(2, result.Count); 
Assert.AreEqual("Kanata", result[0].AdditionalProperties["placeofservicetypename"]); 

Assuming the same entities defined in the Arrange section above,

resourceCacheMock.Verify(sut => sut.GetResources(new Dictionary().....)

The latter does not guarantee we get the correct results but rather the fact that some other resource was invoked, which means we need to know internally how sut.GetResource works.

8.  Minimize the number of asserts in a test.  I would not say a test should be restricted to one assert as this is an artificial constraint which breaks down too often.  Instead, I would advocate for a minimalist approach, encouraging developers to ensure to think about splitting a test if the number of asserts is greater than 3 for example.

9.  Keep tests small or digestable.  It should be relatively straight forward to look at a test and determine what it is doing.  If a colleague cannot decipher the purpose of a test within a couple of minutes, it is overly complex or doing too much.

10.  Test names should be clearly indicate what the test is trying to accomplish.

The EntityMappingService – Application mapping in a Consistent and Disciplined Manner

Here’s a question for you:

How do you map, in a consistent, loosely couple, testable, and extensible manner, objects from your services domain to objects in your application domain?

I have run across various approaches:

  1. Inline in the code using static methods or copy paste code.
  2. Custom code sprinkled all over as needed.
  3. Dedicated service with a well defined interface dedicated to mapping.

Of all these approaches, I am fond of the last one for the following reasons:

  1. Mapping is cross cutting and should be re-usable.
  2. Mapping is not business logic and should be encapsulated away from where such business logic decisions are made.
  3. Mapping is noisy and pollutes the service or view model logic.
  4. Mapping is a single responsibility which can be tested in isolation. The Mappers and mapping repository can be unit tested!

For example, say you have a concept called in invoice represented in two domain; in the RESTFul service domain and your application domain. Once these objects have been retrieved from the service, you will have to map them to corresponding objects in your application domain. To do this, we define a

public class InvoiceToInvoiceDtoMapper : EntityMapperBase
    protected override InvoiceDto DoMap(Invoice source, EntityMappingContext context)
            return new InvoiceDto 
               Number = source.Number,
               AmountDue = source.TotalCharge
               ServiceDate = source.ServiceDate

There will be many of such mappings with each mapping strategy encapsulated by a mapper. How do we manage all of these? We use a central repository of mappers, called the EntiyMapperService.

What is so good about the Entity Mapping Service

It’s responsibility is determining which of these mappers from map from one type to another and delegating the mapping to the mapper in question. It’s interface could look like this:

 public interface IEntityMappingService
        void RegisterEntityMapper(IEntityMapper mapper);

        void UnRegisterEntityMapper(IEntityMapper mapper);

        bool CanMap();

        TTarget Map(object source, EntityMappingContext mappingContext = null);

A naive implementation of such a service could look like this:

 public sealed class EntityMappingService : IEntityMappingService
        private readonly List _entityMappers = new List();

        public EntityMappingService()

        public void RegisterEntityMapper(IEntityMapper mapper)
            if (mapper == null)
                throw new ArgumentNullException("mapper");


        public void RegisterEntityMappers(IEnumerable mappers)
            foreach (var translator in mappers)

        public void UnRegisterEntityMapper(IEntityMapper mapper)
            if (mapper == null)
                throw new ArgumentNullException(nameof(mapper));


        public bool CanMap()
            return CanMap(typeof(TSource), typeof(TTarget));

        public bool CanMap(Type sourceType, Type targetType)
            if (targetType == null)
                throw new ArgumentNullException(nameof(targetType));
            if (sourceType == null)
                throw new ArgumentNullException(nameof(sourceType));
            var entityMapper = FindEntityMapper(targetType, sourceType);
            return entityMapper != null;

        public TTarget Map(object source, EntityMappingContext mappingContext = null)
            return (TTarget)Map(typeof(TTarget), source, mappingContext);
        public object Map(Type targetType, object source, EntityMappingContext mappingContext)
            if (targetType == null)
                throw new ArgumentNullException("targetType");

            if (source == null)
                return null;

            Type sourceType = source.GetType();
            var entityMapper = FindEntityMapper(targetType, sourceType);
            if (entityMapper != null)
                return entityMapper.Map(targetType, source, mappingContext);

            throw new EntityMapperException(string.Format("No mapper is available to perform the operation from {0} to {1}.", sourceType, targetType));

        public IEntityMapper FindEntityMapper(object parameter)
            return _entityMappers.FirstOrDefault(t => t.CanMap(parameter));

        public IEntityMapper FindEntityMapper(Type targetType, Type sourceType)
            return _entityMappers.FirstOrDefault(t => t.CanMap(sourceType, targetType));

        public IEntityMapper FindEntityMapper()
            return _entityMappers.FirstOrDefault(t => t.CanMap(null));

Please note that this mapper does not take into consideration multi-threading nor does it prevent registration of duplicates.

What is wrong with the other approaches

Software is craft and more like an art and there is not a single correct answer. There are better ways of crafting good code and as I have always advised my developers, write code as if you are writing a story. With this mentality, you will find yourself crafting cleaner, robust and more testable code over and over again.

Happy Coding.

Resharper with NUnit does not find local files

This is one of the problems we encounter ever so often but do not take the time to document it.

Say you have some test files in your project and have set their properties to Content and Copy Always.  You are using our favorite Resharper to run your unit tests using the NUnit test runner.  When you attempt to execute this test, you get an error as follows:

 Could not find a part of the path 'C:\Users\knji\AppData\Local\JetBrains\Installations\ReSharperPlatformVs15_427a36eb\TestFiles\PEs\notification.xml'.
 at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
 at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions

When you continue digging, you do realize that your files are indeed present in the bin/Debug folder of your application but Resharper just will not find it. Possible solutions:
1. Turn off Resharper shadow copying. This did not work for us.
2. Instruct your test to explicitly use the current working directory from the TestContext. This worked for us. So here’s the fix. Instead of doing this:

var file = File.ReadAllLines(@"my-relative-folder/some-cool-date.xml");

do this

var file = File.ReadAllLines(@TestContext.CurrentContext.TestDirectory + /"my-relative-folder/some-cool-date.xml");

Happy Coding.

Composing a Client in a Client/Server Architecture Model without Service

I will start by saying this:

If all you have is the functional requirements of the client application and known table structure of the underlying database and the service interface has not yet been established, then let the requirements of the client application drive the service contract.  The structure of the underlying database should not drive this service contract.

Over the course of my career, I have had to architect several solutions against external systems, mostly webservices, which either had not been developed nor had not yet been conceptualized or were in the initial stages of development.

Recently at Phreesia, we had to develop against a web service which was in its initial stages of conception.   At Honeywell’s Global Tracking division , when we migrated the legacy OCC 200, a 20+ years old legacy VMS Search and Rescue application, to the OCC 600, a modern client/server architecture, there was enormous amounts of effort expended in formulating the functional requirements of the desktop application since getting the user interface and all its usability concerns right was of utmost concern to facilitate adoption.

While such efforts was spent on defining the requirements of the client, the requirements of the server were not being developed at the same pace.  Consequently, development of the client started well in advance of the server and in such a case, a decision had to be made on how to compose the Data Access Layer of this desktop application.

We started development of the client side WPF desktop application, by decomposing the application into the three main layers: Presentation Layer Tier, Business Service Layer, Data Access Layer as shown below:

three layered application.png

We had a solid set of functional requirements against the Presentation Layer, which also drove the Middle Layer as well as the DAL. We also had an understanding of the database even though this was not documented and were able to work with the back-end developers to ensure that the contracts we were formulating could be met.  Based on the nature of the application’s use case in Search and Rescue; we are talking about a heavy desktop GIS mapping application, significant amount of functionality was developed before the Web Service came online, creating fake Web Services which implemented the same service contracts.

The Data Access Layer

As the saying goes, this is where the rubber hits the road,  this layer was responsible for retrieving data from the underlying source of truth, our databases and making this available to the application.  It’s purpose was two fold:

  1. Manage all connectivity with the underlying data source:  This was achieved via WCF service constructs.
  2. Encapsulate data retrieval from the application: This was achieved via interfaces and dependency injection.
  3. Map external entities retrieved from the underlying data source to business objects are required by the application: This was achieved via the Service Oriented Architectural and Chain of Responsibility pattern.

This layer was a dedicated set of DLLs which were injected into the application via IoC allowing the front and backend development to continue in a loosely coupled fashion.  The application’s DAL ended up looking like this:

DAL decomposition (1).png

The experiment was a success, after which we had a highly usable and performance WPF desktop application which the end customer has grown to like and adopt.

In another blog, post I will address the issue of mapping from web service entities to the application domain model.  How we did this in a consistent, loosely coupled, testable and extensible manner.

What has been your experiences?

Configuring Asp.NET Core 2 to return Pretty JSON

When creating a WebAPI which returns JSON, it is often times very useful to return formatted JSON, especially if your API does not come with any sort of documentation.

ASP.NET Core 2, makes this easy.  All you have to do in your Web API project is to configure the Json serializer before the application starts and this is done in Startup.cs as follows:

// this method is in the Startup class......

public void ConfigureServices(IServiceCollection services)
 .AddJsonOptions(options => options.SerializerSettings.Formatting = Formatting.Indented );

If your ValuesController returned

["value1", "value2"]

after the above change, it will now return



Once again, happy coding.

Windbg determination of race condition

A report came back from the field indicating that one of our services was not doing what it was designed to do.  Will not accept subscriptions, etc.  No errors were also reported in our Sumo logs.

We decided to take a memory dump of the server and determined there were about 300 threads waiting for something. A quick investigation using Windbg revealed a deadlock or race condition.

First we used !threads to get the following:

0:000> !threads
 ThreadCount: 279
 UnstartedThread: 0
 BackgroundThread: 273
 PendingThread: 0
 DeadThread: 6
 Hosted Runtime: no

Next we used !syncblck to get the following:

0:000> !syncblk
 Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
 33 000000fbfcbf0be8 497 1 000000fc03846780 548 26 000000fbe2724e78 System.Object
 88 000000fbfcbf1458 5 1 000000fc03846780 548 26 000000fbe27248c0 System.Object
 90 000000fbfcbf1368 3 1 000000fc02eeafb0 ad8 23 000000fbe26f80c8 System.Object
 Total 200
 CCW 3
 RCW 5
 ComClassFactory 0
 Free 132

The above table does not readily identify a deadlocked situation.  However, it references two threads worth further investigating to see if the are waiting on locks:  0x548 and 0xad8.  To further probe into each thread, we clicked on the link provided under the “Thread” column in the above table, and once the thread information was retrieved, issued !clrstack as demonstrated below.

Let’s start with thread 0x548.

0:000> ~~[548]s
00007ff9`b0540c6a c3 ret
0:026> !clrstack
OS Thread Id: 0x548 (26)
 Child SP IP Call Site
000000fc0353cca8 00007ff9b0540c6a [GCFrame: 000000fc0353cca8] 
000000fc0353ce18 00007ff9b0540c6a [GCFrame: 000000fc0353ce18] 
000000fc0353ce58 00007ff9b0540c6a [HelperMethodFrame_1OBJ: 000000fc0353ce58] System.Threading.Monitor.Enter(System.Object)
000000fc0353cf50 00007ff948ea067f MyCompany.Redis.Client.RedisClient.CheckForConnect()
000000fc0353cfc0 00007ff948ea05f0 MyCompany.Redis.Client.RedisClient.get_Subscriber()
000000fc0353cff0 00007ff948ea0540 MyCompany.Redis.Client.RedisClient.Subscribe(System.String, System.Action`2)
000000fc0353d060 00007ff948e9fe85 MyCompany.Redis.MessageBus.RedisMessageBus.Subscribe(System.String, System.Action`2)
000000fc0353d138 00007ff9a6bcc29c [StubHelperFrame: 000000fc0353d138] 
000000fc0353d190 00007ff948e9f894 ACompany.Redis.WebSocket.Channel.OnDemandMessageListener.Attach(Integration.OnDemand.WebSocket.Channel.IOnDemandMessageHandler)
000000fc0353d230 00007ff948e9e65b ACompany.Redis.WebSocket.Channel.OnDemandWebSocketHandler.OnOpen()
000000fc0353d380 00007ff948e9e0b6 Microsoft.Web.WebSockets.WebSocketHandler+d__9.MoveNext()
000000fc0353d3e0 00007ff948e9df75 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Web.WebSockets.WebSocketHandler+d__9, Microsoft.WebSockets]](d__9 ByRef)
000000fc0353d490 00007ff948e9dece Microsoft.Web.WebSockets.WebSocketHandler.ProcessWebSocketRequestAsync(System.Web.WebSockets.AspNetWebSocketContext, System.Func`1>)
000000fc0353d550 00007ff949024c4d System.Web.WebSocketPipeline+c__DisplayClass9_0.b__0(System.Object)
000000fc0353d5b0 00007ff9490243f1 System.Web.Util.SynchronizationHelper.SafeWrapCallback(System.Action)
000000fc0353d600 00007ff9490242b6 System.Web.Util.SynchronizationHelper.QueueSynchronous(System.Action)
000000fc0353d660 00007ff949022da1 System.Web.WebSocketPipeline+d__9.MoveNext()
000000fc0353d6f0 00007ff94902268f System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].Start[[System.Web.WebSocketPipeline+d__9, System.Web]](d__9 ByRef)
000000fc0353d7a0 00007ff9490225df System.Web.WebSocketPipeline.ProcessRequestImplAsync()
000000fc0353d860 00007ff949022435 System.Web.WebSocketPipeline.ProcessRequest()
000000fc0353d8b0 00007ff947b6ceb0 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353da60 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353daa0 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0353e2a0 00007ff9a6b07fde [InlinedCallFrame: 000000fc0353e2a0] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0353e2a0 00007ff948157a7e [InlinedCallFrame: 000000fc0353e2a0] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0353e270 00007ff948157a7e DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0353e330 00007ff947b6cdc1 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353e4e0 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353e520 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0353e6f8 00007ff9a6b08233 [ContextTransitionFrame: 000000fc0353e6f8]

This thread is waiting on a lock to be released as indicated by this line at the top of its stack trace:

000000fc0353ce58 00007ff9b0540c6a [HelperMethodFrame_1OBJ: 000000fc0353ce58] System.Threading.Monitor.Enter(System.Object)

Let’s perform a similar exercise for thread 0xad8.  It’s stack trace is as follows:

0:026> ~~[ad8]s
00007ff9`b0540c6a c3 ret
0:023> !clrstack
OS Thread Id: 0xad8 (23)
 Child SP IP Call Site
000000fc0071ccc8 00007ff9b0540c6a [GCFrame: 000000fc0071ccc8] 
000000fc0071cef0 00007ff9b0540c6a [GCFrame: 000000fc0071cef0] 
000000fc0071cf28 00007ff9b0540c6a [HelperMethodFrame: 000000fc0071cf28] System.Threading.Monitor.Enter(System.Object)
000000fc0071d020 00007ff9491de236 Phreesia.Redis.MessageBus.RedisMessageBus.OnRedisConnectionReestablished(System.Object, System.EventArgs)
000000fc0071d110 00007ff947942cd3 [MulticastFrame: 000000fc0071d110] System.EventHandler`1[[System.__Canon, mscorlib]].Invoke(System.Object, System.__Canon)
000000fc0071d170 00007ff9489c6485 ACompany.Redis.Client.RedisClient.InitializeConnection()
000000fc0071d1c0 00007ff948ea06e7 ACompany.Redis.Client.RedisClient.CheckForConnect()
000000fc0071d230 00007ff948ea4da0 ACompany.Redis.Client.RedisClient.get_Database()
000000fc0071d260 00007ff9491b817a ACompany.Redis.Common.RedisOnDemandSubscriptionManager.RemoveSubscription(System.String, System.String, System.String)
000000fc0071d370 00007ff9491b7d73 ACompany.Redis.WebSocket.Channel.OnDemandWebSocketHandler.OnClose()
000000fc0071d4b0 00007ff948e9e50c Microsoft.Web.WebSockets.WebSocketHandler+d__9.MoveNext()
000000fc0071d500 00007ff948e9e3ca Microsoft.Web.WebSockets.WebSocketHandler+d__9.MoveNext()
000000fc0071d560 00007ff948e9df75 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Web.WebSockets.WebSocketHandler+d__9, Microsoft.WebSockets]](d__9 ByRef)
000000fc0071d610 00007ff948e9dece Microsoft.Web.WebSockets.WebSocketHandler.ProcessWebSocketRequestAsync(System.Web.WebSockets.AspNetWebSocketContext, System.Func`1>)
000000fc0071d6d0 00007ff949024c4d System.Web.WebSocketPipeline+c__DisplayClass9_0.b__0(System.Object)
000000fc0071d730 00007ff9490243f1 System.Web.Util.SynchronizationHelper.SafeWrapCallback(System.Action)
000000fc0071d780 00007ff9490242b6 System.Web.Util.SynchronizationHelper.QueueSynchronous(System.Action)
000000fc0071d7e0 00007ff949022da1 System.Web.WebSocketPipeline+d__9.MoveNext()
000000fc0071d870 00007ff94902268f System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].Start[[System.Web.WebSocketPipeline+d__9, System.Web]](d__9 ByRef)
000000fc0071d920 00007ff9490225df System.Web.WebSocketPipeline.ProcessRequestImplAsync()
000000fc0071d9e0 00007ff949022435 System.Web.WebSocketPipeline.ProcessRequest()
000000fc0071da30 00007ff947b6ceb0 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071dbe0 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071dc20 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0071e420 00007ff9a6b07fde [InlinedCallFrame: 000000fc0071e420] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0071e420 00007ff948157a7e [InlinedCallFrame: 000000fc0071e420] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0071e3f0 00007ff948157a7e DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0071e4b0 00007ff947b6cdc1 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071e660 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071e6a0 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0071e878 00007ff9a6b08233 [ContextTransitionFrame: 000000fc0071e878]

Similarly, this thread is waiting for a lock to be released as indicated at the top of its stack trace:

000000fc0071cf28 00007ff9b0540c6a [HelperMethodFrame: 000000fc0071cf28] System.Threading.Monitor.Enter(System.Object)

It gets tricky from this point on with Windbg at one cannot easily identify the lock identifies, which would have helped.  A deadlock means one thread is waiting for a lock to be released, but this thread also has acquiring and is holding a lock which another thread is waiting to be released.

So what can we make from the above?

It starts with thread 0x548 acquiring a lock on its call to RedisClient.Subscribe, let’s call this _redisMessageBusLock, as indicated below:

000000fc0353ce58 00007ff9b0540c6a [HelperMethodFrame_1OBJ: 000000fc0353ce58] System.Threading.Monitor.Enter(System.Object)
000000fc0353cf50 00007ff948ea067f Phreesia.Redis.Client.RedisClient.CheckForConnect()
000000fc0353cfc0 00007ff948ea05f0 Phreesia.Redis.Client.RedisClient.get_Subscriber()
000000fc0353cff0 00007ff948ea0540 Phreesia.Redis.Client.RedisClient.Subscribe(System.String, System.Action`2)   // we acquire _redisMessageBus lock here

Simultaneously, thread 0xad8, had first attempted to check for the Redis connection state in its call to CheckForConnect().

Servicing of this call, requires acquisition of a lock, we will call _redisClientLock. We acquire the lock, attempt to fix the Redis connection, if there is a problem, raise an event when the connection has been re-established, then release the lock.

In this case, there was an issue with the Redis connection, so we acquired the lock, _redisClientLock, fixed the connection and then raised the event OnRedisConnectionReestablished, while still holding the lock, _redisClientLock.

000000fc0071cf28 00007ff9b0540c6a [HelperMethodFrame: 000000fc0071cf28] System.Threading.Monitor.Enter(System.Object)    // we are waiting for redisMessageBusLock to be released
000000fc0071d020 00007ff9491de236 Phreesia.Redis.MessageBus.RedisMessageBus.OnRedisConnectionReestablished(System.Object, System.EventArgs)
000000fc0071d110 00007ff947942cd3 [MulticastFrame: 000000fc0071d110] System.EventHandler`1[[System.__Canon, mscorlib]].Invoke(System.Object, System.__Canon)
000000fc0071d170 00007ff9489c6485 Phreesia.Redis.Client.RedisClient.InitializeConnection()
000000fc0071d1c0 00007ff948ea06e7 Phreesia.Redis.Client.RedisClient.CheckForConnect()     // we acquired redisClientLock here

Since the event handling is synchronous, the same thread which raises the event, handles, it.  However, part of handling the event requires acquisition of redisMessageBusLock, which was already acquired by thread ox548.

Based on the above, our lock state looks like this:

Thread       LockAcquired           LockWaitingFor          
0xad8        _redisClientLock        _redisMessageBusLock     
0x548        _redisMessageBusLock    _redisClientLock

which clearly demonstrates out deadlock.

Happy debugging.