Redis Cluster is a built-in feature of Redis that
provides automatic sharding, replication, and high availability. It replaces
the previously used Sentinels for these purposes. Redis Cluster is designed to
automatically split your dataset among multiple instances and provide some
level of availability during partitions, allowing operations to continue even
when some instances (especially masters) fail or are unable to communicate with
the majority of nodes in the cluster.
However, Redis Cluster may stop operating in the event of larger failures, such as when the majority of master instances are unavailable. If a master and slave fail simultaneously, the cluster cannot continue normal operations, although a workaround is to add more nodes or create asymmetry in the cluster to
automatically change the cluster layout.
According to the Redis Cluster documentation, a "minimal cluster" that functions as expected should have at least 3 master nodes. However, for a more suitable setup with higher availability, it is recommended to have at least 6 nodes, with three masters and three slaves, where each master has a
corresponding slave.
It's important to note that Redis Cluster has some limitations, including lack of support for NATted environments or environments where IP addresses or TCP ports are remapped, such as in Docker. Additionally, not all client libraries may fully support Redis Cluster.
This article will provide a step-by-step guide on how to set up a Redis Cluster (with Cluster-Mode Disabled) in CentOS 8. It will cover the installation of Redis, configuration of the cluster nodes, creation of the cluster, and testing of the cluster failover.
Note:
For this guide, we will use fresh/empty Redis instances to run the cluster mode. The cluster mode will not work with some configurations done in the first two guides of our Redis Series, particularly when the replica of the parameter is being used.
Prerequisites:
Test
Environment Setup
Redis Master1: 10.42.0.247
Redis Master2: 10.42.0.197
Redis Master3: 10.42.0.132
Redis Slave1: 10.42.0.200
Redis Slave2: 10.42.0.21
Redis Slave3: 10.42.0.34
Redis Cluster Logical Diagram
Our setup has
3 read/write master nodes and 3 read-only replica nodes, each master having one
replica, so three shards contain all of the cluster’s data in each node. An
application API
or CLI
client can write only to the master nodes but read from any node in the
cluster.
Step 1:
Installing Redis on All Nodes
1. Log into all
the instances via SSH, then run the following command to install the
Redis
module using the DNF
package manager as shown.
# dnf module install redis
2. Next, start
the Redis
service, enable it to automatically start at system boot and check its status
to verify that it is running (verify the service on all the 6 instances):
# systemctl start redis# systemctl enable redis# systemctl status redis
Step 2:
Configuring Redis Instances on all Nodes
3. This section
describes how to configure the Redis cluster nodes. Remember to perform the
configurations here on all the nodes.
Use the /etc/redis.conf
configuration file to configure the Redis server. As a recommended practice, create a
backup of the original file before editing it using a command-line
text editor of your choice.
# cp /etc/redis.conf /etc/redis.conf.orig# vi /etc/redis.conf
4. Next, find
the following configuration parameters and edit their values as shown. The bind
parameter sets the interface of the Redis server will listen on, set its value
to the instance LAN IP. Remove the 127.0.0.1 because we realized leaving it there
slows down the process of cluster creation, particularly the stage of joining
the cluster.
bind 10.42.0.247
Then set the protected-mode
to no to allow
connections from the other instances on the cluster.
protected-mode no
The port parameter defines the port
the Redis server will listen on for connections, the default is 6379.
This is the data port for communicating with clients
Set Redis Listen Interface and Port
5. The next set
of parameters will enable the cluster mode and set some of its useful features.
The cluster-enabled
parameter, when set to yes,
activates the cluster mode.
cluster-enabled yes
Next, the cluster-config-file
parameter sets the name of a cluster node’s cluster configuration file (e.g nodes-6379.conf).
The file is created in the working directory (default is /var/lib/redis
defined using the dir parameter) and is not user editable.
cluster-config-file nodes-6379.conf
The next
useful cluster option is cluster-node-timeout, it is used to set the
maximum amount of time in milliseconds an instance can be unavailable for it to
be considered in a failure state. A value of 15000 is
equivalent to 15
seconds.
cluster-node-timeout 15000
Set Cluster Node Timeout
6. We also need
to enable Redis persistence on disk. We can use one of the persistence modes,
that is the Append
Only File (AOF): it logs (in the file appendonly.aof
created under the working directory) every write operation successfully
received by the server. The data will be played during the server startup to
reconstruct the original dataset.
To enable it,
set the appendonly
parameter to yes.
appendonly yes
Set Persistence Options
7. After making
all the changes, restart the Redis service on all the nodes to apply the recent
changes.
# systemctl restart redis
8. At this
point, every cluster node should now have an ID. You can check
this in the logfile located at /var/log/redis/redis.log.
# cat /var/log/redis/redis.log
Check Cluster Node Log File
9. Next, open
port 6397
and 16379
on all the instances. The later port is used for the cluster bus (a node-to-node
communication channel using a binary protocol). This is a basic requirement for
the Redis cluster TCP connections.
# firewall-cmd --zone=public --permanent --add-port=6379/tcp # firewall-cmd --zone=public --permanent --add-port=16379/tcp # firewall-cmd --reload
Step 3:
Creating the Redis Cluster
10. To create
the cluster, use the redis-cli command-line client as follows. The --cluster create enables cluster creation and --cluster-replicas 1 means create one replica per
master.
For our setup
which has 6 nodes, we will have 3 masters and 3 slaves.
Note that the
first 6 nodes will be considered masters (M)
and the next three will be considered slaves (S).
The first slave i.e 10.42.0.200:6379 replicates the first master i.e 10.42.0.247:6379,
the second slave replicates the second master, in that order.
The following
command is formatted in a way that the outcome will represent our logical setup
above.
# redis-cli --cluster create 10.42.0.247:6379 10.42.0.197:6379 10.42.0.132:6379 10.42.0.200:6379 10.42.0.21:6379 10.42.0.34:6379 --cluster-replicas 1
Create Redis Cluster
11. Once the
cluster creation is successful, run the following command on any host (specify
its IP address using the -h
flag) to list all cluster nodes.
# redis-cli -h 10.42.0.247 -p 6379 cluster nodes
You should be
able to see all the cluster nodes, with the slaves indicating their masters, as
shown in the following screenshot.
Check All Cluster Nodes on Any Node
The different
fields are in this order: node ID, IP address:port, flags, last ping sent, last
pong received, configuration epoch, link-state, slots (for masters).
Step 4:
Testing Redis Cluster Failover
12. In this
section, we will demonstrate how to test a cluster failover. First, let’s take
note of the masters.
# redis-cli -h 10.42.0.247 -p 6379 cluster nodes | grep master
List Redis Cluster Masters
Also, take
note of the Redis slaves.
# redis-cli -h 10.42.0.247 -p 6379 cluster nodes | grep slave
List All Redis Cluster Slaves
13. Next, let’s
stop the Redis service on one of the master nodes e.g 10.42.0.197
and check all master nodes in the cluster.
# systemctl stop redis# redis-cli -h 10.42.0.247 -p 6379 cluster nodes | grep master
From the
following screenshot, you can see that the node 10.42.0.197:6367
is in fail state and its slave 10.42.0.21:6379 has been promoted to master
status.
Check Cluster Failover Status
14. Now let’s
start the Redis
service once again on the failed node and check all the masters in the cluster.
# systemctl start redis# redis-cli -h 10.42.0.247 -p 6379 cluster nodes | grep master
Check All Redis Cluster Master Status
Also, check
the cluster slaves to confirm that the failed master is now a slave.
# redis-cli -h 10.42.0.247 -p 6379 cluster nodes | grep slave
Check All Redis Cluster Slaves
Step 5:
Testing Data Replication Across the Redis Cluster
15. This last
section explains how to verify cluster data replication. We will create a key
and value on one of the masters, then try to read it from all the cluster nodes
as follows. Use the -c
switch to enable cluster support under redis-cli utility and access data in
cluster mode.
# redis-cli -c -h 10.42.0.247 -p 6379 set name 'TecMint.com'# redis-cli -c -h 10.42.0.247 -p 6379 get name# redis-cli -c -h 10.42.0.21 -p 6379 get name# redis-cli -c -h 10.42.0.132 -p 6379 get name# redis-cli -c -h 10.42.0.200 -p 6379 get name# redis-cli -c -h 10.42.0.197 -p 6379 get name# redis-cli -c -h 10.42.0.34 -p 6379 get name
Verify Redis Cluster Data Replication.
In addition to the basic
configuration settings covered in this article, there are many other
configuration parameters available in the /etc/redis.conf file that can be used
to fine-tune the performance, security, and behavior of Redis Cluster. These
parameters are well-documented in the Redis documentation, and understanding
their usage can help optimize the performance and reliability of your Redis
Cluster deployment.
Some of the commonly used
configuration parameters in Redis Cluster include settings related to network
interfaces, ports, timeouts, authentication, memory management, persistence,
and replication. By carefully configuring these parameters based on your
specific requirements and use case, you can customize the behavior of Redis
Cluster to meet the needs of your application.
It's important to thoroughly
review the Redis documentation and understand the implications of each
configuration parameter before making any changes to the /etc/redis.conf file.
Incorrect configuration settings can impact the performance, stability, and
security of your Redis Cluster. Therefore, it's recommended to test any
configuration changes in a controlled environment before applying them to a
production Redis Cluster.
In conclusion, Redis Cluster is a
powerful and reliable solution for achieving automatic sharding, replication,
and high availability in Redis. By leveraging the various configuration
parameters available in the /etc/redis.conf file, you can fine-tune the
behavior of Redis Cluster to meet the specific requirements of your application
and ensure optimal performance.
No comments:
Post a Comment