Be on lookout for StackExchange Redis ConnectionMultiplexer ConnectionFailed event misfires

We are using the StackExchange.Redis ConnectionMultiplexer class to manage our connections to Redis.  Without clear guidance from the documentation, we have attempted to create our own retry strategy with this client with code which looks like this:

 

  public RedisClient()
 : this(GetConnectionString())
 {
 }
 
 public RedisClient(string connectionString)
 {
 _logger = LogServiceProvider.Instance.GetLogger(GetType());
 _connectionString = connectionString;
 MaxRetryAttempts = 10;
 DelayBeforeRetryInMillisecs = 500;
 InitializeConnection();
 }

 private void InitializeConnection()
 {
 _logger.Info("Initializing a connection to the Redis cluster. ");
 bool isReconnectionAttempt = false;

 if (_connectionMultiplexer != null)
 {
 Debug.WriteLine("disposing " + _connectionMultiplexer.GetHashCode());
 _connectionMultiplexer.ConnectionFailed -= HandleConnectionFailedEvent;

 // test this change.....
 _connectionMultiplexer.Close(false);
 isReconnectionAttempt = true;
 _logger.Info("This is reconnection attempt to the Redis cluster.");
 }

 _connectionMultiplexer = ConnectionMultiplexer.Connect(_connectionString);
 _needConnect = !_connectionMultiplexer.IsConnected;
 _connectionMultiplexer.ConnectionFailed += HandleConnectionFailedEvent;

 Debug.WriteLine("created " + _connectionMultiplexer.GetHashCode());

 if (!_needConnect && isReconnectionAttempt)
 {
 _logger.Info("Reconnection to the Redis cluster was succeeded.");
 RaiseRedisConnectionReestablished();
 }

 if (!_needConnect)
 {
 _logger.Info("Connection is successfully established to the Redis cluster.");
 }
 else
 {
 _logger.Error("Cannot establish a connection to the Redis cluster. ");
 }
 }

 private void HandleConnectionFailedEvent(object sender, ConnectionFailedEventArgs args)
 {
 // we could potentially receive multiple of these events so we need to be be careful
 Debug.WriteLine( "received connection failure event from " + sender.GetHashCode());
 Debug.WriteLine(" multiplexer id is " + _connectionMultiplexer.GetHashCode());

 // There's a known issue with the Redis ConnectionMultiplexer which prevents if from 
 // completely releasing an event handler even after it has been disconnected and disposed.
 // Se we need the following line of code.

 if (sender.GetHashCode() != _connectionMultiplexer.GetHashCode())
 return;

 _logger.Error("Connection to the Redis cluster has failed with the following event arg:", args.Exception);

 RaiseRedisConnectionFailed(args.Exception);
 AttemptReconnection(1);
 }

 private void AttemptReconnection(int trial = 1)
 {
 if (trial == MaxRetryAttempts)
 {
 _logger.Info("Have attempted to re-connect to the Redis cluster 3 times. Aborting.");
 
 return;
 }

 if (_connectionMultiplexer.IsConnected)
 {
 _logger.Info("Connetion to Redis cluster is no longer required.");
 return;
 }

 _logger.InfoFormat("Attempting reconnection attempt {0} to the Redis cluster.", trial);
 RedisReconnectionAttempt(this, new RedisReconnectionAttemptEventArgs(trial));

 // wait for a few seconds and then try to reconnect.....
 Thread.Sleep(DelayBeforeRetryInMillisecs);
 InitializeConnection();

 trial = trial + 1;
 TotalNumberOfReconnectionAttempts = TotalNumberOfReconnectionAttempts + 1;
 AttemptReconnection(trial);
 
 }

When a network drop is simulated, the ConnectionFailed event is fired as expected. When this happens, an attempt is made to dispose of the current instance of the ConnectionMultiplexer object and create a new one. We do this to avoid the situation where attempting to access the current ConnectoinMultiplexer instance throws an exception indicating the object has already been disposed.

So we dispose the current instance, or atleast we think we do and create a new one. Yet somehow, the original instance which experienced the network drop, but is now disposed, still manages to fire ConnectionFailed events even though we are no longer supposed to be listerning.  After all, we unsubscribed to the event as indicated in the InitializeConnection method above.

I could not get an answer from StackOverflow. I also could not determine the cause just by looking at the source code for this ConnectionMultiplexer class.  What I could do, however, is put in a small hack to ensure these multiple noisy events are ignored, unless they are coming from the correct ConnectionMultiplexer instance.

I even have test to ensure my hack works.

Happy coding.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s