Something, possibly a network connection or cluster failure happened, requiring the Redis cluster to switch around the masters. The default port for Redis cluster masters is 6379. However, after the switch, Redis masters where listening on port 6380.
All our connection strings pointing to the Redis cluster do not explicitly specify a port, which means our services are all trying to publish and subscribe to Redis masters on port 6379, which no longer were there after a port switch.
This information was obtained by connecting to a Redis node and executing the info command.
C:\dev\tools\redis>redis-cli -h 18.104.22.168
os:Linux 2.6.32-642.1.1.el6.x86_64 x86_64
This information indicates that this node, which was previously believed to be a master has now been relegated to a slave node, For pub-sub in Redis to work, the connection strings should specify either the exact ip and ports to the Redis master nodes or all the ip addresses and ports of all the nodes in the Redis cluster.
This problem was manifested as failure of Redis to recognize a subscription to a channel when the the appropriate client started. This client subscribes to a Redis channel during startup. However, while monitoring activities on all Redis nodes using the “monitor” command, it was observed that when the client is restarted, there was no subscription being registered to Redis for the channel. Also, when the internal RESTful services published a message a Redis, this activity was also not being recorded while monitoring the three “master” nodes in the cluster.
This is with the original conneciton strings specifying IP addresses of the three Redis boxes without ports as follows:
<add name=”redis” connectionString=”22.214.171.124,126.96.36.199,188.8.131.52″ />
After running the Redis info command and determining that there were no masters listening on the default port of 6379, and explicitly specifying the port on which the masters were listening to, all services were able to establish a connection with Redis.
So, here’s an interim solution which works until we come up with a comprehensive strategy:
All Redis connection strings should include all the nodes (master and slaves) with explicit specification of ip addresses and ports. For example, these settings as configured in the RESTFul and WebSocket services look like this:
<add name=”redis” connectionString=”184.108.40.206:6379,220.127.116.11:6379,18.104.22.168:6379,22.214.171.124:6380,126.96.36.199:6380,188.8.131.52:6380″ />
While researching into this, it was also discovered that Redis does provide a channel called “__Booksleeve_MasterChanged”, which provides a change notification when master configuration changes. Clients can subscribe to messages on this channel to determine a cluster topology change and act accordingly. The list of available channels currently open on a Redis node can be retrieved using command “pubsub channels”.