Effect of Redis cluster master/slave Reconfiguration

Something, possibly a network connection or cluster failure happened, requiring the Redis cluster to switch around the masters. The default port for Redis cluster masters is 6379. However, after the switch, Redis masters where listening on port 6380.

All our connection strings pointing to the Redis cluster do not explicitly specify a port, which means our services are all trying to publish and subscribe to Redis masters on port 6379, which no longer were there after a port switch.

This information was obtained by connecting to a Redis node and executing the info command.

C:\dev\tools\redis>redis-cli -h 1.1.2.3
1.1.2.3:6379> info

  1. Server
    redis_version:3.0.7
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:46ce43dec62732a2
    redis_mode:cluster
    os:Linux 2.6.32-642.1.1.el6.x86_64 x86_64
    arch_bits:64
    multiplexing_api:epoll
    gcc_version:4.4.7
    process_id:30733
    run_id:fa7f530a07abfeae43f45a45e6f5ec03fa738864
    tcp_port:6379
    uptime_in_seconds:5263766
    uptime_in_days:60
    hz:10
    lru_clock:12363344
    config_file:/etc/redis/6379/6379.conf
  1. Clients
    connected_clients:52
    client_longest_output_list:0
    client_biggest_input_buf:0
    blocked_clients:0
  1. Memory
    used_memory:2397912
    used_memory_human:2.29M
    used_memory_rss:9166848
    used_memory_peak:4133760
    used_memory_peak_human:3.94M
    used_memory_lua:36864
    mem_fragmentation_ratio:3.82
    mem_allocator:jemalloc-3.6.0
  1. Persistence
    loading:0
    rdb_changes_since_last_save:0
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1471980906
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:0
    rdb_current_bgsave_time_sec:-1
    aof_enabled:1
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:0
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok
    aof_last_write_status:ok
    aof_current_size:80890
    aof_base_size:1568
    aof_pending_rewrite:0
    aof_buffer_length:0
    aof_rewrite_buffer_length:0
    aof_pending_bio_fsync:0
    aof_delayed_fsync:0
  1. Stats
    total_connections_received:712
    total_commands_processed:6673948
    instantaneous_ops_per_sec:1
    total_net_input_bytes:265947707
    total_net_output_bytes:327706638
    instantaneous_input_kbps:0.05
    instantaneous_output_kbps:0.31
    rejected_connections:0
    sync_full:1
    sync_partial_ok:0
    sync_partial_err:0
    expired_keys:0
    evicted_keys:0
    keyspace_hits:27
    keyspace_misses:0
    pubsub_channels:7
    pubsub_patterns:0
    latest_fork_usec:53252
    migrate_cached_sockets:0
  1. Replication
    role:slave
    master_host:1.1.2.4
    master_port:6380
    master_link_status:up
    master_last_io_seconds_ago:4
    master_sync_in_progress:0
    slave_repl_offset:8171386
    slave_priority:100
    slave_read_only:1
    connected_slaves:0
    master_repl_offset:0
    repl_backlog_active:0
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:6439780
    repl_backlog_histlen:1048576
  1. CPU
    used_cpu_sys:3938.52
    used_cpu_user:2750.70
    used_cpu_sys_children:4.43
    used_cpu_user_children:0.37
  1. Cluster
    cluster_enabled:1
  1. Keyspace
    db0:keys=12,expires=0,avg_ttl=0

This information indicates that this node, which was previously believed to be a master has now been relegated to a slave node, For pub-sub in Redis to work, the connection strings should specify either the exact ip and ports to the Redis master nodes or all the ip addresses and ports of all the nodes in the Redis cluster.

This problem was manifested as failure of Redis to recognize a subscription to a channel when the the appropriate client started.  This client subscribes to a Redis channel during startup.  However, while monitoring activities on all Redis nodes using the “monitor” command, it was observed that when the client is restarted, there was no subscription being registered to Redis for the channel. Also, when the internal RESTful services published a message a Redis, this activity was also not being recorded while monitoring the three “master” nodes in the cluster.
This is with the original conneciton strings specifying IP addresses of the three Redis boxes without ports as follows:

<add name=”redis” connectionString=”1.1.2.3,1.1.2.4,1.1.2.5″ />
After running the Redis info command and determining that there were no masters listening on the default port of 6379, and explicitly specifying the port on which the masters were listening to, all services were able to establish a connection with Redis.

So, here’s an interim solution which works until we come up with a comprehensive strategy:

All Redis connection strings should include all the nodes (master and slaves) with explicit specification of ip addresses and ports. For example, these settings as configured in the RESTFul and WebSocket services look like this:

<add name=”redis” connectionString=”1.1.2.3:6379,1.1.2.4:6379,1.1.2.5:6379,1.1.2.3:6380,1.1.2.4:6380,1.1.2.5:6380″ />

While researching into this, it was also discovered that Redis does provide a channel called “__Booksleeve_MasterChanged”, which provides a change notification when master configuration changes. Clients can subscribe to messages on this channel to determine a cluster topology change and act accordingly. The list of available channels currently open on a Redis node can be retrieved using command “pubsub channels”.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s