Category Archives: Uncategorized

Resharper with NUnit does not find local files

This is one of the problems we encounter ever so often but do not take the time to document it.

Say you have some test files in your project and have set their properties to Content and Copy Always.  You are using our favorite Resharper to run your unit tests using the NUnit test runner.  When you attempt to execute this test, you get an error as follows:

 Could not find a part of the path 'C:\Users\knji\AppData\Local\JetBrains\Installations\ReSharperPlatformVs15_427a36eb\TestFiles\PEs\notification.xml'.
 at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
 at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions

When you continue digging, you do realize that your files are indeed present in the bin/Debug folder of your application but Resharper just will not find it. Possible solutions:
1. Turn off Resharper shadow copying. This did not work for us.
2. Instruct your test to explicitly use the current working directory from the TestContext. This worked for us. So here’s the fix. Instead of doing this:

var file = File.ReadAllLines(@"my-relative-folder/some-cool-date.xml");

do this

var file = File.ReadAllLines(@TestContext.CurrentContext.TestDirectory + /"my-relative-folder/some-cool-date.xml");

Happy Coding.

Advertisements

Windbg determination of race condition

A report came back from the field indicating that one of our services was not doing what it was designed to do.  Will not accept subscriptions, etc.  No errors were also reported in our Sumo logs.

We decided to take a memory dump of the server and determined there were about 300 threads waiting for something. A quick investigation using Windbg revealed a deadlock or race condition.

First we used !threads to get the following:

0:000> !threads
 ThreadCount: 279
 UnstartedThread: 0
 BackgroundThread: 273
 PendingThread: 0
 DeadThread: 6
 Hosted Runtime: no

Next we used !syncblck to get the following:

0:000> !syncblk
 Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
 33 000000fbfcbf0be8 497 1 000000fc03846780 548 26 000000fbe2724e78 System.Object
 88 000000fbfcbf1458 5 1 000000fc03846780 548 26 000000fbe27248c0 System.Object
 90 000000fbfcbf1368 3 1 000000fc02eeafb0 ad8 23 000000fbe26f80c8 System.Object
 -----------------------------
 Total 200
 CCW 3
 RCW 5
 ComClassFactory 0
 Free 132

The above table does not readily identify a deadlocked situation.  However, it references two threads worth further investigating to see if the are waiting on locks:  0x548 and 0xad8.  To further probe into each thread, we clicked on the link provided under the “Thread” column in the above table, and once the thread information was retrieved, issued !clrstack as demonstrated below.

Let’s start with thread 0x548.

0:000> ~~[548]s
ntdll!NtWaitForMultipleObjects+0xa:
00007ff9`b0540c6a c3 ret
0:026> !clrstack
OS Thread Id: 0x548 (26)
 Child SP IP Call Site
000000fc0353cca8 00007ff9b0540c6a [GCFrame: 000000fc0353cca8] 
000000fc0353ce18 00007ff9b0540c6a [GCFrame: 000000fc0353ce18] 
000000fc0353ce58 00007ff9b0540c6a [HelperMethodFrame_1OBJ: 000000fc0353ce58] System.Threading.Monitor.Enter(System.Object)
000000fc0353cf50 00007ff948ea067f MyCompany.Redis.Client.RedisClient.CheckForConnect()
000000fc0353cfc0 00007ff948ea05f0 MyCompany.Redis.Client.RedisClient.get_Subscriber()
000000fc0353cff0 00007ff948ea0540 MyCompany.Redis.Client.RedisClient.Subscribe(System.String, System.Action`2)
000000fc0353d060 00007ff948e9fe85 MyCompany.Redis.MessageBus.RedisMessageBus.Subscribe(System.String, System.Action`2)
000000fc0353d138 00007ff9a6bcc29c [StubHelperFrame: 000000fc0353d138] 
000000fc0353d190 00007ff948e9f894 ACompany.Redis.WebSocket.Channel.OnDemandMessageListener.Attach(Integration.OnDemand.WebSocket.Channel.IOnDemandMessageHandler)
000000fc0353d230 00007ff948e9e65b ACompany.Redis.WebSocket.Channel.OnDemandWebSocketHandler.OnOpen()
000000fc0353d380 00007ff948e9e0b6 Microsoft.Web.WebSockets.WebSocketHandler+d__9.MoveNext()
000000fc0353d3e0 00007ff948e9df75 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Web.WebSockets.WebSocketHandler+d__9, Microsoft.WebSockets]](d__9 ByRef)
000000fc0353d490 00007ff948e9dece Microsoft.Web.WebSockets.WebSocketHandler.ProcessWebSocketRequestAsync(System.Web.WebSockets.AspNetWebSocketContext, System.Func`1>)
000000fc0353d550 00007ff949024c4d System.Web.WebSocketPipeline+c__DisplayClass9_0.b__0(System.Object)
000000fc0353d5b0 00007ff9490243f1 System.Web.Util.SynchronizationHelper.SafeWrapCallback(System.Action)
000000fc0353d600 00007ff9490242b6 System.Web.Util.SynchronizationHelper.QueueSynchronous(System.Action)
000000fc0353d660 00007ff949022da1 System.Web.WebSocketPipeline+d__9.MoveNext()
000000fc0353d6f0 00007ff94902268f System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].Start[[System.Web.WebSocketPipeline+d__9, System.Web]](d__9 ByRef)
000000fc0353d7a0 00007ff9490225df System.Web.WebSocketPipeline.ProcessRequestImplAsync()
000000fc0353d860 00007ff949022435 System.Web.WebSocketPipeline.ProcessRequest()
000000fc0353d8b0 00007ff947b6ceb0 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353da60 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353daa0 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0353e2a0 00007ff9a6b07fde [InlinedCallFrame: 000000fc0353e2a0] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0353e2a0 00007ff948157a7e [InlinedCallFrame: 000000fc0353e2a0] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0353e270 00007ff948157a7e DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0353e330 00007ff947b6cdc1 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353e4e0 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0353e520 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0353e6f8 00007ff9a6b08233 [ContextTransitionFrame: 000000fc0353e6f8]

This thread is waiting on a lock to be released as indicated by this line at the top of its stack trace:

000000fc0353ce58 00007ff9b0540c6a [HelperMethodFrame_1OBJ: 000000fc0353ce58] System.Threading.Monitor.Enter(System.Object)

Let’s perform a similar exercise for thread 0xad8.  It’s stack trace is as follows:

0:026> ~~[ad8]s
ntdll!NtWaitForMultipleObjects+0xa:
00007ff9`b0540c6a c3 ret
0:023> !clrstack
OS Thread Id: 0xad8 (23)
 Child SP IP Call Site
000000fc0071ccc8 00007ff9b0540c6a [GCFrame: 000000fc0071ccc8] 
000000fc0071cef0 00007ff9b0540c6a [GCFrame: 000000fc0071cef0] 
000000fc0071cf28 00007ff9b0540c6a [HelperMethodFrame: 000000fc0071cf28] System.Threading.Monitor.Enter(System.Object)
000000fc0071d020 00007ff9491de236 Phreesia.Redis.MessageBus.RedisMessageBus.OnRedisConnectionReestablished(System.Object, System.EventArgs)
000000fc0071d110 00007ff947942cd3 [MulticastFrame: 000000fc0071d110] System.EventHandler`1[[System.__Canon, mscorlib]].Invoke(System.Object, System.__Canon)
000000fc0071d170 00007ff9489c6485 ACompany.Redis.Client.RedisClient.InitializeConnection()
000000fc0071d1c0 00007ff948ea06e7 ACompany.Redis.Client.RedisClient.CheckForConnect()
000000fc0071d230 00007ff948ea4da0 ACompany.Redis.Client.RedisClient.get_Database()
000000fc0071d260 00007ff9491b817a ACompany.Redis.Common.RedisOnDemandSubscriptionManager.RemoveSubscription(System.String, System.String, System.String)
000000fc0071d370 00007ff9491b7d73 ACompany.Redis.WebSocket.Channel.OnDemandWebSocketHandler.OnClose()
000000fc0071d4b0 00007ff948e9e50c Microsoft.Web.WebSockets.WebSocketHandler+d__9.MoveNext()
000000fc0071d500 00007ff948e9e3ca Microsoft.Web.WebSockets.WebSocketHandler+d__9.MoveNext()
000000fc0071d560 00007ff948e9df75 System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Web.WebSockets.WebSocketHandler+d__9, Microsoft.WebSockets]](d__9 ByRef)
000000fc0071d610 00007ff948e9dece Microsoft.Web.WebSockets.WebSocketHandler.ProcessWebSocketRequestAsync(System.Web.WebSockets.AspNetWebSocketContext, System.Func`1>)
000000fc0071d6d0 00007ff949024c4d System.Web.WebSocketPipeline+c__DisplayClass9_0.b__0(System.Object)
000000fc0071d730 00007ff9490243f1 System.Web.Util.SynchronizationHelper.SafeWrapCallback(System.Action)
000000fc0071d780 00007ff9490242b6 System.Web.Util.SynchronizationHelper.QueueSynchronous(System.Action)
000000fc0071d7e0 00007ff949022da1 System.Web.WebSocketPipeline+d__9.MoveNext()
000000fc0071d870 00007ff94902268f System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, mscorlib]].Start[[System.Web.WebSocketPipeline+d__9, System.Web]](d__9 ByRef)
000000fc0071d920 00007ff9490225df System.Web.WebSocketPipeline.ProcessRequestImplAsync()
000000fc0071d9e0 00007ff949022435 System.Web.WebSocketPipeline.ProcessRequest()
000000fc0071da30 00007ff947b6ceb0 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071dbe0 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071dc20 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0071e420 00007ff9a6b07fde [InlinedCallFrame: 000000fc0071e420] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0071e420 00007ff948157a7e [InlinedCallFrame: 000000fc0071e420] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0071e3f0 00007ff948157a7e DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000fc0071e4b0 00007ff947b6cdc1 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071e660 00007ff947b6c3e4 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000fc0071e6a0 00007ff947b6b8ab DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000fc0071e878 00007ff9a6b08233 [ContextTransitionFrame: 000000fc0071e878]

Similarly, this thread is waiting for a lock to be released as indicated at the top of its stack trace:

000000fc0071cf28 00007ff9b0540c6a [HelperMethodFrame: 000000fc0071cf28] System.Threading.Monitor.Enter(System.Object)

It gets tricky from this point on with Windbg at one cannot easily identify the lock identifies, which would have helped.  A deadlock means one thread is waiting for a lock to be released, but this thread also has acquiring and is holding a lock which another thread is waiting to be released.

So what can we make from the above?

It starts with thread 0x548 acquiring a lock on its call to RedisClient.Subscribe, let’s call this _redisMessageBusLock, as indicated below:

000000fc0353ce58 00007ff9b0540c6a [HelperMethodFrame_1OBJ: 000000fc0353ce58] System.Threading.Monitor.Enter(System.Object)
000000fc0353cf50 00007ff948ea067f Phreesia.Redis.Client.RedisClient.CheckForConnect()
000000fc0353cfc0 00007ff948ea05f0 Phreesia.Redis.Client.RedisClient.get_Subscriber()
000000fc0353cff0 00007ff948ea0540 Phreesia.Redis.Client.RedisClient.Subscribe(System.String, System.Action`2)   // we acquire _redisMessageBus lock here

Simultaneously, thread 0xad8, had first attempted to check for the Redis connection state in its call to CheckForConnect().

Servicing of this call, requires acquisition of a lock, we will call _redisClientLock. We acquire the lock, attempt to fix the Redis connection, if there is a problem, raise an event when the connection has been re-established, then release the lock.

In this case, there was an issue with the Redis connection, so we acquired the lock, _redisClientLock, fixed the connection and then raised the event OnRedisConnectionReestablished, while still holding the lock, _redisClientLock.

000000fc0071cf28 00007ff9b0540c6a [HelperMethodFrame: 000000fc0071cf28] System.Threading.Monitor.Enter(System.Object)    // we are waiting for redisMessageBusLock to be released
000000fc0071d020 00007ff9491de236 Phreesia.Redis.MessageBus.RedisMessageBus.OnRedisConnectionReestablished(System.Object, System.EventArgs)
000000fc0071d110 00007ff947942cd3 [MulticastFrame: 000000fc0071d110] System.EventHandler`1[[System.__Canon, mscorlib]].Invoke(System.Object, System.__Canon)
000000fc0071d170 00007ff9489c6485 Phreesia.Redis.Client.RedisClient.InitializeConnection()
000000fc0071d1c0 00007ff948ea06e7 Phreesia.Redis.Client.RedisClient.CheckForConnect()     // we acquired redisClientLock here

Since the event handling is synchronous, the same thread which raises the event, handles, it.  However, part of handling the event requires acquisition of redisMessageBusLock, which was already acquired by thread ox548.

Based on the above, our lock state looks like this:

Thread       LockAcquired           LockWaitingFor          
0xad8        _redisClientLock        _redisMessageBusLock     
0x548        _redisMessageBusLock    _redisClientLock

which clearly demonstrates out deadlock.

Happy debugging.

HTTP 301: The requested resource SHOULD be accessed through returned URI in Location Header.

Ever attempted to test a public API using, what you think is the correct credentials and query parameters only to get the above error?

For example, I tried the SumoLogic Search Jobs API with what I thought was the correct credentials and query parameters and kept getting this response:

{
 "status" : 301,
 "id" : "OLXXO-HINM6-3BXX7",
 "code" : "moved",
 "message" : "The requested resource SHOULD be accessed through returned URI in Location Header."
}

I even passed in an invalid URL and received the same response with a different “id”.

Quick solution to this problem is to run the curl command with the -v (verbose) option and it will spit out the location you are supposed to target as shown below:

< Cache-control: no-cache="set-cookie"
< Content-Type: application/json; charset=ISO-8859-1
< Date: Sat, 05 Nov 2016 03:59:46 GMT
< Location: https://api.us2.sumologic.com/api/v1/search/jobs?query=nnnnn

In my case, I was targeting:

https://api.sumologic.com/api/v1/search/jobs

instead of

https://api.us2.sumologic.com/api/v1/search/jobs

Sprint story work break down – how do you do it?

Every agile team appears to break down the tasks involved in completing a sprint story item a little different  Also, every team  appears to have a different perspective on the Srum Definition of Done.  My perspective aligns closely with one of my previous manager’s definition:

A story is considered done, if it is shelvable, packageable and shippable.  Basically at the toss of a hat, it can be deployed to production or made available to end customers with all associated artifacts including supporting documentation.

This is the philosophy which I use as guideline when confronted with the task of splitting down sprint stories.  And why so?

Let’s start by stating the assumption that we have a story with a well written, understood and itemized set of user, technical or deployment requirements. These requirements should drive both development, test and documentation.  Or should they?

When a story is considered done, it should be possible to validate that each requirement was met, including existence of a shippable component to end customers, whoever they may be.   Such that at the end of the sprint, the team collectively as a whole, including PM and additional stakeholders should be able to validate that each of the requirements listed in the story is appropriately captured in the resulting artifact.

There are various distinct developmental tasks involved in taking a marketable idea from concept to market.  These tasks include:

  • Design : captures some high level design activity including UX work for UI related stories.
  • Development : everyone knows what this is all about.  Yes, this is the task which captures all coding effort.
  • QA: everyone also should know what this is about.   Someone has to validate the output from the Development tasks to ensure it does meet user requirements specified in the story. An interesting point of contention I have run into involves the source document from which QA should author test cases.  One school of thought says test cases should be driven from user requirements in the story and another says  from the output of the design task. This alone is an interesting topic on its own.
  • Deployment : this task captures work such as creating of chef scripts or other activities dedicated to ensuring code makes it from the build machine into our production servers.  Some companies use DevOps engineers for this.  Others get the same developers to do it all.  Again another interesting topic on its own.
  • Documentation: almost a task which never gets the time of day, especially if product is targeted to internal customers.

On one team, we had a hard and fast rule stating that EVERY story should be broken down into each of the aforementioned subtasks.  However, as with a lot of things in life such hard and fast rules do not apply and often times completely break down, forcing the team into a “process oriented as opposed to goal oriented mindset” as one of my team members  succinctly puts it.  I see these tasks as mere guidelines. Completing some stories will require all of these subtasks while others will not and it is up to the Scrum master in consultation with the team to use wise judgement to make this decision.

Should these distinct activities be captured by individual JIRA subtasks?  I personally think so.  Individual subtasks allow distinct teams to start the work in parrallel, enabling early engagement by all respective teams, possibly allowing for faster delivery of the feature.  Of course, the assumption here is we have distinct teams responsible for each facet of the development workflow.  If we have a single developer responsible for orchestrating each of the aforementioned phases, it is no necessary to explicitly break down the story into such tasks even though it may be worth capturing the effort required during each stage of the workflow.

What do you other Scrum Masters do?

Should models have simple methods

I recently ran into an interesting conversation with some members of my team.

We have a class to define configuration of a communication infrastructure.  While utilizing an instance of this class with a factory,  we decided to create several methods in the model to allow us perform simple checks, such as IsChannelEnabled as illustrated below:


public class ClientCommunicationConfiguration
 {

public string Version { get; set; }
 public string Description { get; set; }
 public bool IsEnabled { get; set; }
 public List<ChannelConfiguration> Channels { get; set; }

public bool HasChannels()
 {
 return Channels != null && Channels.Any();
 }

public bool IsChannelEnabled(string channelName)
 {
 if (!HasChannels())
 return false;

return Channels.Any(c => string.Compare(c.Name, channelName,StringComparison.OrdinalIgnoreCase) == 0 &amp;amp;&amp;amp; c.IsEnabled);
 }

public bool IsMethodEnabled(string messageName)
 {
 return Channels
 .Where(c => c.HasMessages() && c.IsEnabled)
 .SelectMany(c => c.Messages)
 .Any(m => string.Compare(m.Name, messageName, StringComparison.OrdinalIgnoreCase) == 0 &amp;amp;&amp;amp; m.IsEnabled);

}
 }

A member of my team made a good argument stating that models should only expose properties.  It is not the responsibility of the model to make these kinds of decisions.  His argument is that these methods should reside in the factory class or in some other management entity which contains business logic to make these determinations.  It is definitely a good point although I made the following counter arguments:

  1. Every class in .NET comes with 3 methods, ToString(), GetHashCode and Equals.  Therefore models are not pure in that sense.
  2. We can encapsulate minimal logic in a model to allow one make certain determinations that are inherent in the model’s definition.
  3. Encapsulating such logic in the model makes the model testable as well, otherwise we have to create entities just to wrap such logic and ensure testability.
  4. There is no such thing as a strict model without methods.  Models are simply serializable objects. They can still have methods exposed to do simply checks, return data based on their internal state and just do simple validation.
  5. This validation code can be re-used.
  6. OO means a piece of data should contain properties and methods.

One advantage I see with not having these methods in model is simply a matter of purity. Otherwise, I do not see a realistic reason why models should not expose methods.

What are your thoughts?

 

Effect of Redis cluster master/slave Reconfiguration

Something, possibly a network connection or cluster failure happened, requiring the Redis cluster to switch around the masters. The default port for Redis cluster masters is 6379. However, after the switch, Redis masters where listening on port 6380.

All our connection strings pointing to the Redis cluster do not explicitly specify a port, which means our services are all trying to publish and subscribe to Redis masters on port 6379, which no longer were there after a port switch.

This information was obtained by connecting to a Redis node and executing the info command.

C:\dev\tools\redis>redis-cli -h 1.1.2.3
1.1.2.3:6379> info

  1. Server
    redis_version:3.0.7
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:46ce43dec62732a2
    redis_mode:cluster
    os:Linux 2.6.32-642.1.1.el6.x86_64 x86_64
    arch_bits:64
    multiplexing_api:epoll
    gcc_version:4.4.7
    process_id:30733
    run_id:fa7f530a07abfeae43f45a45e6f5ec03fa738864
    tcp_port:6379
    uptime_in_seconds:5263766
    uptime_in_days:60
    hz:10
    lru_clock:12363344
    config_file:/etc/redis/6379/6379.conf
  1. Clients
    connected_clients:52
    client_longest_output_list:0
    client_biggest_input_buf:0
    blocked_clients:0
  1. Memory
    used_memory:2397912
    used_memory_human:2.29M
    used_memory_rss:9166848
    used_memory_peak:4133760
    used_memory_peak_human:3.94M
    used_memory_lua:36864
    mem_fragmentation_ratio:3.82
    mem_allocator:jemalloc-3.6.0
  1. Persistence
    loading:0
    rdb_changes_since_last_save:0
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1471980906
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:0
    rdb_current_bgsave_time_sec:-1
    aof_enabled:1
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:0
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok
    aof_last_write_status:ok
    aof_current_size:80890
    aof_base_size:1568
    aof_pending_rewrite:0
    aof_buffer_length:0
    aof_rewrite_buffer_length:0
    aof_pending_bio_fsync:0
    aof_delayed_fsync:0
  1. Stats
    total_connections_received:712
    total_commands_processed:6673948
    instantaneous_ops_per_sec:1
    total_net_input_bytes:265947707
    total_net_output_bytes:327706638
    instantaneous_input_kbps:0.05
    instantaneous_output_kbps:0.31
    rejected_connections:0
    sync_full:1
    sync_partial_ok:0
    sync_partial_err:0
    expired_keys:0
    evicted_keys:0
    keyspace_hits:27
    keyspace_misses:0
    pubsub_channels:7
    pubsub_patterns:0
    latest_fork_usec:53252
    migrate_cached_sockets:0
  1. Replication
    role:slave
    master_host:1.1.2.4
    master_port:6380
    master_link_status:up
    master_last_io_seconds_ago:4
    master_sync_in_progress:0
    slave_repl_offset:8171386
    slave_priority:100
    slave_read_only:1
    connected_slaves:0
    master_repl_offset:0
    repl_backlog_active:0
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:6439780
    repl_backlog_histlen:1048576
  1. CPU
    used_cpu_sys:3938.52
    used_cpu_user:2750.70
    used_cpu_sys_children:4.43
    used_cpu_user_children:0.37
  1. Cluster
    cluster_enabled:1
  1. Keyspace
    db0:keys=12,expires=0,avg_ttl=0

This information indicates that this node, which was previously believed to be a master has now been relegated to a slave node, For pub-sub in Redis to work, the connection strings should specify either the exact ip and ports to the Redis master nodes or all the ip addresses and ports of all the nodes in the Redis cluster.

This problem was manifested as failure of Redis to recognize a subscription to a channel when the the appropriate client started.  This client subscribes to a Redis channel during startup.  However, while monitoring activities on all Redis nodes using the “monitor” command, it was observed that when the client is restarted, there was no subscription being registered to Redis for the channel. Also, when the internal RESTful services published a message a Redis, this activity was also not being recorded while monitoring the three “master” nodes in the cluster.
This is with the original conneciton strings specifying IP addresses of the three Redis boxes without ports as follows:

<add name=”redis” connectionString=”1.1.2.3,1.1.2.4,1.1.2.5″ />
After running the Redis info command and determining that there were no masters listening on the default port of 6379, and explicitly specifying the port on which the masters were listening to, all services were able to establish a connection with Redis.

So, here’s an interim solution which works until we come up with a comprehensive strategy:

All Redis connection strings should include all the nodes (master and slaves) with explicit specification of ip addresses and ports. For example, these settings as configured in the RESTFul and WebSocket services look like this:

<add name=”redis” connectionString=”1.1.2.3:6379,1.1.2.4:6379,1.1.2.5:6379,1.1.2.3:6380,1.1.2.4:6380,1.1.2.5:6380″ />

While researching into this, it was also discovered that Redis does provide a channel called “__Booksleeve_MasterChanged”, which provides a change notification when master configuration changes. Clients can subscribe to messages on this channel to determine a cluster topology change and act accordingly. The list of available channels currently open on a Redis node can be retrieved using command “pubsub channels”.

 

A refresher on .NET Binding Redirect

What exactly does this line in the .csproj mean?

<Reference Include="Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
 <HintPath>..\..\packages\Newtonsoft.Json.1.0.2\lib\4.0\Newtonsoft.Json.dll</HintPath>
 <Private>True</Private>
 </Reference>

First, let break down each line at a time:

<Reference Include=

<Reference Include=   is an XML tag denoting an assembly reference into a project.

The text within the include tag        Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL    is the fully qualified name of the .NET assembly.

<HintPath>  denotes a location where Visual Studio will first attempt to look for the referenced DLL before starting to look in its probing paths.

As you can see, there are two file versions here, 1.0.2 and 4.5.0.0 which typically lead to confusion.  However, when having a discussion around assemblies, it is best to stick with the one used in the fully qualified name, which in this case is 4.5.0.0.  This is the version we use in the rest of this article.

So, why does all of this matter?

Different projects in a solution can use different versions of a DLL.  For example, one project could rely v4.5.0.0 of Newtonsoft.Json while another one could rely on v6.0.0.0. However, when both projects are built to formulate the solution package, which one will be used, if both have the correct version of these assemblies in their hint paths?

As we all know, or we should, there cannot be two DLLs with the same name within a folder.  So when our solution is deployed, there will be only one Newtonsoft.Json in the installation folder.  The version deployed, will depend on the last project that was built and its output copied into the installation folder for our application.

But, what if we deployed v4.5.0.0 of the assembly?  What would the assembly which relied on 6.0.0.0 of this assembly do, when it has to resolve its types?

 

This is where binding redirect for .NET comes in.  In simple terms binding redirect instructs the .NET runtime on what version of an assembly to use if it cannot find the one that was specified in the assembly manifest.  This is a configuration in app.config or web.config as typically looks like this:

<runtime>
<dependentAssembly>
 <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
 <bindingRedirect oldVersion="0.0.0.0-8.0.0.0" newVersion="4.5.0.0" />
 </dependentAssembly>
 </assemblyBinding>
 </runtime>

Basically, we are saying, her .NET runtime, if you are attempting to resolve an assembly called Newtonsoft.Json which has any version from 0.0.0.0 to 8.0.0.0, please look for an use Newtonsoft.Json version 4.5.0.0.

This will all work like magic but you better be sure Newtonsoft.Json 4.5.0.0 deployed alongside with your application and that this version is fully compatible with all versions within the range 0.0.0.0 to 8.0.0.0.

Happing Coding.