Monthly Archives: August 2016

Effect of Redis cluster master/slave Reconfiguration

Something, possibly a network connection or cluster failure happened, requiring the Redis cluster to switch around the masters. The default port for Redis cluster masters is 6379. However, after the switch, Redis masters where listening on port 6380.

All our connection strings pointing to the Redis cluster do not explicitly specify a port, which means our services are all trying to publish and subscribe to Redis masters on port 6379, which no longer were there after a port switch.

This information was obtained by connecting to a Redis node and executing the info command.

C:\dev\tools\redis>redis-cli -h 1.1.2.3
1.1.2.3:6379> info

  1. Server
    redis_version:3.0.7
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:46ce43dec62732a2
    redis_mode:cluster
    os:Linux 2.6.32-642.1.1.el6.x86_64 x86_64
    arch_bits:64
    multiplexing_api:epoll
    gcc_version:4.4.7
    process_id:30733
    run_id:fa7f530a07abfeae43f45a45e6f5ec03fa738864
    tcp_port:6379
    uptime_in_seconds:5263766
    uptime_in_days:60
    hz:10
    lru_clock:12363344
    config_file:/etc/redis/6379/6379.conf
  1. Clients
    connected_clients:52
    client_longest_output_list:0
    client_biggest_input_buf:0
    blocked_clients:0
  1. Memory
    used_memory:2397912
    used_memory_human:2.29M
    used_memory_rss:9166848
    used_memory_peak:4133760
    used_memory_peak_human:3.94M
    used_memory_lua:36864
    mem_fragmentation_ratio:3.82
    mem_allocator:jemalloc-3.6.0
  1. Persistence
    loading:0
    rdb_changes_since_last_save:0
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1471980906
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:0
    rdb_current_bgsave_time_sec:-1
    aof_enabled:1
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:0
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok
    aof_last_write_status:ok
    aof_current_size:80890
    aof_base_size:1568
    aof_pending_rewrite:0
    aof_buffer_length:0
    aof_rewrite_buffer_length:0
    aof_pending_bio_fsync:0
    aof_delayed_fsync:0
  1. Stats
    total_connections_received:712
    total_commands_processed:6673948
    instantaneous_ops_per_sec:1
    total_net_input_bytes:265947707
    total_net_output_bytes:327706638
    instantaneous_input_kbps:0.05
    instantaneous_output_kbps:0.31
    rejected_connections:0
    sync_full:1
    sync_partial_ok:0
    sync_partial_err:0
    expired_keys:0
    evicted_keys:0
    keyspace_hits:27
    keyspace_misses:0
    pubsub_channels:7
    pubsub_patterns:0
    latest_fork_usec:53252
    migrate_cached_sockets:0
  1. Replication
    role:slave
    master_host:1.1.2.4
    master_port:6380
    master_link_status:up
    master_last_io_seconds_ago:4
    master_sync_in_progress:0
    slave_repl_offset:8171386
    slave_priority:100
    slave_read_only:1
    connected_slaves:0
    master_repl_offset:0
    repl_backlog_active:0
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:6439780
    repl_backlog_histlen:1048576
  1. CPU
    used_cpu_sys:3938.52
    used_cpu_user:2750.70
    used_cpu_sys_children:4.43
    used_cpu_user_children:0.37
  1. Cluster
    cluster_enabled:1
  1. Keyspace
    db0:keys=12,expires=0,avg_ttl=0

This information indicates that this node, which was previously believed to be a master has now been relegated to a slave node, For pub-sub in Redis to work, the connection strings should specify either the exact ip and ports to the Redis master nodes or all the ip addresses and ports of all the nodes in the Redis cluster.

This problem was manifested as failure of Redis to recognize a subscription to a channel when the the appropriate client started.  This client subscribes to a Redis channel during startup.  However, while monitoring activities on all Redis nodes using the “monitor” command, it was observed that when the client is restarted, there was no subscription being registered to Redis for the channel. Also, when the internal RESTful services published a message a Redis, this activity was also not being recorded while monitoring the three “master” nodes in the cluster.
This is with the original conneciton strings specifying IP addresses of the three Redis boxes without ports as follows:

<add name=”redis” connectionString=”1.1.2.3,1.1.2.4,1.1.2.5″ />
After running the Redis info command and determining that there were no masters listening on the default port of 6379, and explicitly specifying the port on which the masters were listening to, all services were able to establish a connection with Redis.

So, here’s an interim solution which works until we come up with a comprehensive strategy:

All Redis connection strings should include all the nodes (master and slaves) with explicit specification of ip addresses and ports. For example, these settings as configured in the RESTFul and WebSocket services look like this:

<add name=”redis” connectionString=”1.1.2.3:6379,1.1.2.4:6379,1.1.2.5:6379,1.1.2.3:6380,1.1.2.4:6380,1.1.2.5:6380″ />

While researching into this, it was also discovered that Redis does provide a channel called “__Booksleeve_MasterChanged”, which provides a change notification when master configuration changes. Clients can subscribe to messages on this channel to determine a cluster topology change and act accordingly. The list of available channels currently open on a Redis node can be retrieved using command “pubsub channels”.

 

Advertisements

A refresher on .NET Binding Redirect

What exactly does this line in the .csproj mean?

<Reference Include="Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
 <HintPath>..\..\packages\Newtonsoft.Json.1.0.2\lib\4.0\Newtonsoft.Json.dll</HintPath>
 <Private>True</Private>
 </Reference>

First, let break down each line at a time:

<Reference Include=

<Reference Include=   is an XML tag denoting an assembly reference into a project.

The text within the include tag        Newtonsoft.Json, Version=4.5.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL    is the fully qualified name of the .NET assembly.

<HintPath>  denotes a location where Visual Studio will first attempt to look for the referenced DLL before starting to look in its probing paths.

As you can see, there are two file versions here, 1.0.2 and 4.5.0.0 which typically lead to confusion.  However, when having a discussion around assemblies, it is best to stick with the one used in the fully qualified name, which in this case is 4.5.0.0.  This is the version we use in the rest of this article.

So, why does all of this matter?

Different projects in a solution can use different versions of a DLL.  For example, one project could rely v4.5.0.0 of Newtonsoft.Json while another one could rely on v6.0.0.0. However, when both projects are built to formulate the solution package, which one will be used, if both have the correct version of these assemblies in their hint paths?

As we all know, or we should, there cannot be two DLLs with the same name within a folder.  So when our solution is deployed, there will be only one Newtonsoft.Json in the installation folder.  The version deployed, will depend on the last project that was built and its output copied into the installation folder for our application.

But, what if we deployed v4.5.0.0 of the assembly?  What would the assembly which relied on 6.0.0.0 of this assembly do, when it has to resolve its types?

 

This is where binding redirect for .NET comes in.  In simple terms binding redirect instructs the .NET runtime on what version of an assembly to use if it cannot find the one that was specified in the assembly manifest.  This is a configuration in app.config or web.config as typically looks like this:

<runtime>
<dependentAssembly>
 <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
 <bindingRedirect oldVersion="0.0.0.0-8.0.0.0" newVersion="4.5.0.0" />
 </dependentAssembly>
 </assemblyBinding>
 </runtime>

Basically, we are saying, her .NET runtime, if you are attempting to resolve an assembly called Newtonsoft.Json which has any version from 0.0.0.0 to 8.0.0.0, please look for an use Newtonsoft.Json version 4.5.0.0.

This will all work like magic but you better be sure Newtonsoft.Json 4.5.0.0 deployed alongside with your application and that this version is fully compatible with all versions within the range 0.0.0.0 to 8.0.0.0.

Happing Coding.