kubernetes生产实践之redis-cluster

##方案一自定义yaml文件安装redis cluster

背景

在Kubernetes中部署Redis集群面临挑战，因为每个Redis实例都依赖于一个配置文件，该文件可以跟踪其他集群实例及其角色。为此，需要结合使用Kubernetes StatefulSets和PersistentVolumes实现。
redis cluster架构示意图：

创建StatefulSet yaml文件

---apiVersion: v1kind: ConfigMapmetadata:  name: redis-clusterdata:  update.sh: |    #!/bin/sh    REDIS_NODES="/data/nodes.conf"    sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}    exec "$@"  redis.conf: |+    bind 0.0.0.0    cluster-enabled yes    cluster-require-full-coverage no    cluster-node-timeout 30000    cluster-config-file /data/nodes.conf    cluster-migration-barrier 1    appendonly yes    protected-mode no---apiVersion: apps.kruise.io/v1beta1# apiVersion: apps/v1kind: StatefulSetmetadata:  name: redis-clusterspec:  serviceName: redis-cluster  replicas: 6  selector:    matchLabels:      app: redis-cluster  template:    metadata:      labels:        app: redis-cluster    spec:      containers:      - name: redis        image: redis:6.2.1-alpine        ports:        - containerPort: 6379          name: client        - containerPort: 16379          name: gossip        command: ["/conf/update.sh", "redis-server", "/conf/redis.conf"]        env:        - name: POD_IP          valueFrom:            fieldRef:              fieldPath: status.podIP        volumeMounts:        - name: conf          mountPath: /conf          readOnly: false        - name: data          mountPath: /data          readOnly: false      volumes:      - name: conf        configMap:          name: redis-cluster          defaultMode: 0755  volumeClaimTemplates:  - metadata:      name: data    spec:      accessModes: [ "ReadWriteOnce" ]      resources:        requests:          storage: 100Gi      storageClassName: rbd

注释：

cluster-migration-barrier 那些分配后仍然剩余migration barrier个从节点的主节点才会触发节点分配，而不是分配前有migration barrier个从节点的主节点就会触发节点分配，默认是1，生产环境建议维持默认值protected-mode no 参数是为了禁止外网访问redis，如果启用了，则只能够通过lookback ip（127.0.0.1）访问Redis，如果外网访问redis，会报出异常。apiVersion: apps.kruise.io/v1beta1 控制器这里使用kruise提供的Advanced StatefulSet，如果集群没有安装kruise，可以使用 apps/v1

安装redis-cluster

[root@qd01-stop-k8s-master001 redis]# kubectl apply -f install-redis.yamlconfigmap/redis-cluster createdstatefulset.apps.kruise.io/redis-cluster created[root@qd01-stop-k8s-master001 redis]# kubectl get po -n opNAME              READY   STATUS    RESTARTS   AGEredis-cluster-0   1/1     Running   0          3m26sredis-cluster-1   1/1     Running   0          3m14sredis-cluster-2   1/1     Running   0          2m54sredis-cluster-3   1/1     Running   0          2m23sredis-cluster-4   1/1     Running   0          2m14sredis-cluster-5   1/1     Running   0          114s

创建redis-cluster service

---apiVersion: v1kind: Servicemetadata:  name: redis-cluster  namespace: opspec:  type: ClusterIP  ports:  - port: 6379    targetPort: 6379    name: client  - port: 16379    targetPort: 16379    name: gossip  selector:    app: redis-cluster

[root@qd01-stop-k8s-master001 redis]# kubectl apply -f redis-svc.ymlservice/redis-cluster created[root@qd01-stop-k8s-master001 redis]# kubectl get svc -n opNAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGEredis-cluster   ClusterIP   10.97.197.224   <none>        6379/TCP,16379/TCP   9s测试能后连通[root@qd01-stop-k8s-master001 redis]# telnet  10.97.197.224 6379Trying 10.97.197.224...Connected to 10.97.197.224.Escape character is '^]'.

初始化redis-cluster

执行如下命令，获取到pod IP，然后使用redis-cli --cluster创建集群

[root@qd01-stop-k8s-master001 redis]# kubectl -n op exec -it redis-cluster-0 -- redis-cli --cluster create --cluster-replicas 1 $(kubectl -n op get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 {end}')>>> Performing hash slots allocation on 6 nodes...Master[0] -> Slots 0 - 5460Master[1] -> Slots 5461 - 10922Master[2] -> Slots 10923 - 16383Adding replica 100.88.43.67:6379 to 100.64.147.152:6379Adding replica 100.113.170.5:6379 to 100.98.174.217:6379Adding replica 100.64.147.153:6379 to 100.80.158.227:6379M: b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379   slots:[0-5460] (5461 slots) masterM: 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379   slots:[5461-10922] (5462 slots) masterM: 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379   slots:[10923-16383] (5461 slots) masterS: b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379   replicates 5389ace495b68eeac85370d6783648dff68f2fb6S: 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379   replicates b47b27a3dbddf3fc1370cbe14ae753f4fce20b04S: e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379   replicates 09543217c903350e963fc4fdf4acb73f8a1b7f8bCan I set the above configuration? (type 'yes' to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join..>>> Performing Cluster Check (using node 100.64.147.152:6379)M: b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379   slots:[0-5460] (5461 slots) master   1 additional replica(s)S: 0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379   slots: (0 slots) slave   replicates b47b27a3dbddf3fc1370cbe14ae753f4fce20b04M: 09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379   slots:[5461-10922] (5462 slots) master   1 additional replica(s)M: 5389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379   slots:[10923-16383] (5461 slots) master   1 additional replica(s)S: e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379   slots: (0 slots) slave   replicates 09543217c903350e963fc4fdf4acb73f8a1b7f8bS: b1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379   slots: (0 slots) slave   replicates 5389ace495b68eeac85370d6783648dff68f2fb6[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.

验证集群信息

[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-0 -- redis-cli cluster infocluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:6cluster_my_epoch:1cluster_stats_messages_ping_sent:178cluster_stats_messages_pong_sent:181cluster_stats_messages_sent:359cluster_stats_messages_ping_received:176cluster_stats_messages_pong_received:178cluster_stats_messages_meet_received:5cluster_stats_messages_received:359[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-0 -- redis-cli cluster nodes0113f4668ec2f3ca2e9470c44bd5faab532b0936 100.88.43.67:6379@16379 slave b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 0 1615348311156 1 connected09543217c903350e963fc4fdf4acb73f8a1b7f8b 100.98.174.217:6379@16379 master - 0 1615348314162 2 connected 5461-10922b47b27a3dbddf3fc1370cbe14ae753f4fce20b04 100.64.147.152:6379@16379 myself,master - 0 1615348312000 1 connected 0-54605389ace495b68eeac85370d6783648dff68f2fb6 100.80.158.227:6379@16379 master - 0 1615348312000 3 connected 10923-16383e1e2f18ae66c79f1943390beabb59613abbad38a 100.113.170.5:6379@16379 slave 09543217c903350e963fc4fdf4acb73f8a1b7f8b 0 1615348313160 2 connectedb1f39714c006ae55b12b18e6537303d7a00e1704 100.64.147.153:6379@16379 slave 5389ace495b68eeac85370d6783648dff68f2fb6 0 1615348312158 3 connected

从输出可以看到，集群总共6个节点，三主三从

##方案二使用kubeDB安装redis

安装kubeDB

1、安装KubeDB
获取AppsCode License https://license-issuer.appscode.com/
下载KubeDB charts https://github.com/appscode/charts/tree/master/stable/kubedb-community

[root@qd01-stop-k8s-master001 kubedb-community]# unzip kubedb-community-v0.16.2.tgz[root@qd01-stop-k8s-master001 kubedb-community]# cd kubedb-community[root@qd01-stop-k8s-master001 kubedb-community]# ls -altotal 96drwxr-xr-x 4 root root   158 Mar 10 15:26 .drwxr-xr-x 3 root root    66 Mar 10 15:24 ..-rw-r--r-- 1 root root   351 Feb 16 09:55 Chart.yamldrwxr-xr-x 2 root root    28 Mar 10 15:24 ci-rw-r--r-- 1 root root   493 Feb 16 09:55 doc.yaml-rw-r--r-- 1 root root   353 Feb 16 09:55 .helmignore-rw-r--r-- 1 root root 24422 Feb 16 09:55 README.mddrwxr-xr-x 2 root root  4096 Mar 10 15:24 templates-rw-r--r-- 1 root root 47437 Feb 16 09:55 values.openapiv3_schema.yaml-rw-r--r-- 1 root root  5230 Feb 16 09:55 values.yaml

修改values.yaml，把License文件放到kubedb-community目录下
2、使用helm安装

[root@qd01-stop-k8s-master001 kubedb-community]# helm install kubedb-community --namespace kube-system --set-file license=./kubedb-community-license.txt -f values.yaml  .NAME: kubedb-communityLAST DEPLOYED: Wed Mar 10 15:38:59 2021NAMESPACE: kube-systemSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:To verify that KubeDB has started, run:  kubectl get deployment --namespace kube-system -l "app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"Now install/upgrade appscode/kubedb-catalog chart.To install, run:  helm install kubedb-catalog appscode/kubedb-catalog --version v0.16.2 --namespace kube-systemTo upgrade, run:  helm upgrade kubedb-catalog appscode/kubedb-catalog --version v0.16.2 --namespace kube-system执行如下命令查看是否安装完成[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get deployment --namespace kube-system -l "app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community"NAME               READY   UP-TO-DATE   AVAILABLE   AGEkubedb-community   1/1     1            1           38s等待crds注册成功[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get crds -l app.kubernetes.io/name=kubedb -wNAME                                       CREATED ATelasticsearches.kubedb.com                 2021-03-10T07:39:42Zelasticsearchversions.catalog.kubedb.com   2021-03-10T07:39:45Zetcds.kubedb.com                           2021-03-10T07:39:42Zetcdversions.catalog.kubedb.com            2021-03-10T07:39:45Zmemcacheds.kubedb.com                      2021-03-10T07:39:43Zmemcachedversions.catalog.kubedb.com       2021-03-10T07:39:45Zmongodbs.kubedb.com                        2021-03-10T07:39:43Zmongodbversions.catalog.kubedb.com         2021-03-10T07:39:45Zmysqls.kubedb.com                          2021-03-10T07:39:43Zmysqlversions.catalog.kubedb.com           2021-03-10T07:39:46Zperconaxtradbs.kubedb.com                  2021-03-10T07:39:43Zperconaxtradbversions.catalog.kubedb.com   2021-03-10T07:39:46Zpgbouncers.kubedb.com                      2021-03-10T07:39:44Zpgbouncerversions.catalog.kubedb.com       2021-03-10T07:39:46Zpostgreses.kubedb.com                      2021-03-10T07:39:44Zpostgresversions.catalog.kubedb.com        2021-03-10T07:39:46Zproxysqls.kubedb.com                       2021-03-10T07:39:44Zproxysqlversions.catalog.kubedb.com        2021-03-10T07:39:46Zredises.kubedb.com                         2021-03-10T07:39:45Zredisversions.catalog.kubedb.com           2021-03-10T07:39:46Z

3、安装KubeDB Catalog
同样，先下载https://github.com/appscode/charts/tree/master/stable/kubedb-catalog

[root@qd01-stop-k8s-master001 kubedb-catalog]# tar -zxf kubedb-catalog-v0.16.2.tgz[root@qd01-stop-k8s-master001 kubedb-catalog]# cd kubedb-catalog[root@qd01-stop-k8s-master001 kubedb-catalog]# ls -altotal 24drwxr-xr-x  3 root root  148 Mar 10 15:48 .drwxr-xr-x  3 root root   28 Mar 10 15:48 ..-rw-r--r--  1 root root  321 Jan 26 20:08 Chart.yaml-rw-r--r--  1 root root  467 Jan 26 20:08 doc.yaml-rw-r--r--  1 root root  353 Jan 26 20:08 .helmignore-rw-r--r--  1 root root 3195 Jan 26 20:08 README.mddrwxr-xr-x 12 root root  188 Mar 10 15:48 templates-rw-r--r--  1 root root  744 Jan 26 20:08 values.openapiv3_schema.yaml-rw-r--r--  1 root root 1070 Jan 26 20:08 values.yaml[root@qd01-stop-k8s-master001 kubedb-catalog]# helm install kubedb-catalog --namespace kube-system -f values.yaml  .NAME: kubedb-catalogLAST DEPLOYED: Wed Mar 10 15:50:50 2021NAMESPACE: kube-systemSTATUS: deployedREVISION: 1TEST SUITE: None

使用kubedb安装redis

1、先看下官方给的redis生命周期示意图

kubedb安装redis支持如下特性

Features    AvailabilityClustering  ✓Instant Backup  ✗Scheduled Backup    ✗Persistent Volume   ✓Initialize using Snapshot   ✗Initialize using Script ✗Custom Configuration    ✓Using Custom docker image   ✓Builtin Prometheus Discovery    ✓Using Prometheus operator   ✓

2、查看支持的版本

[root@qd01-stop-k8s-master001 kubedb-catalog]# kubectl get redisversionsNAME       VERSION   DB_IMAGE                DEPRECATED   AGE4.0.11     4.0.11    kubedb/redis:4.0.11                  15m4.0.6-v2   4.0.6     kubedb/redis:4.0.6-v2                15m5.0.3-v1   5.0.3     kubedb/redis:5.0.3-v1                15m6.0.6      6.0.6     kubedb/redis:6.0.6                   15m

3、编辑yaml安装文件
可以参照https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml
这里选择安装6.0.6这个版本，我的集群storageClassName: "rbd"，请根据实际修改
如果想自定义redis.conf,请参考https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml

apiVersion: kubedb.com/v1alpha2kind: Redismetadata:  name: redis-cluster  namespace: opspec:  version: 6.0.6  mode: Cluster  cluster:    master: 3    replicas: 1  storageType: Durable  storage:    resources:      requests:        storage: 1Gi    storageClassName: "rbd"    accessModes:      - ReadWriteOnce

执行安装

[root@qd01-stop-k8s-master001 kubedb-community]# kubectl apply -f redis-cluster.yamlredis.kubedb.com/redis-cluster created安装完成，可以如下查看[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get rd,po -n opNAME                             VERSION   STATUS         AGEredis.kubedb.com/redis-cluster   6.0.6     Provisioning   6m55sNAME                         READY   STATUS    RESTARTS   AGEpod/redis-cluster-shard0-0   1/1     Running   0          6m54spod/redis-cluster-shard0-1   1/1     Running   0          6m18spod/redis-cluster-shard1-0   1/1     Running   0          5m38spod/redis-cluster-shard1-1   1/1     Running   0          5m1spod/redis-cluster-shard2-0   1/1     Running   0          4m30spod/redis-cluster-shard2-1   1/1     Running   0          4m8s[root@qd01-stop-k8s-master001 redis]# kubectl get svc -n opNAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGEredis-cluster        ClusterIP   10.97.197.224   <none>        6379/TCP   5h16mredis-cluster-pods   ClusterIP   None            <none>        6379/TCP   17m

4、验证集群

[root@qd01-stop-k8s-master001 kubedb-community]# kubectl get pods -n  op  -o jsonpath='{range.items[*]}{.metadata.name} ---------- {.status.podIP}:6379{"\t\n"}{end}' | grep redisredis-cluster-shard0-0 ---------- 100.64.147.156:6379redis-cluster-shard0-1 ---------- 100.98.174.218:6379redis-cluster-shard1-0 ---------- 100.126.252.204:6379redis-cluster-shard1-1 ---------- 100.113.170.6:6379redis-cluster-shard2-0 ---------- 100.107.55.69:6379redis-cluster-shard2-1 ---------- 100.78.230.4:6379[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-shard0-0  -- redis-cli cluster infocluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:3cluster_my_epoch:1cluster_stats_messages_ping_sent:864cluster_stats_messages_pong_sent:882cluster_stats_messages_sent:1746cluster_stats_messages_ping_received:879cluster_stats_messages_pong_received:864cluster_stats_messages_meet_received:3cluster_stats_messages_received:1746[root@qd01-stop-k8s-master001 redis]# kubectl -n op  exec -it redis-cluster-shard0-0  -- redis-cli cluster nodes1895cb4b9c31b848666c61000e502f55a29a8255 100.64.147.155:6379@16379 master - 0 1615365162008 2 connected 5461-1092230bdbf2ca37001774498a9b935afbc1cd2ce389c 100.126.252.203:6379@16379 slave 2c06092fafa99e0158e39e6237a04fed25be3550 0 1615365163000 1 connected9b2cfbd5c1b417121d410141b6da9512ad29ce3c 100.78.230.3:6379@16379 slave e83446c368839c5fdccf5f70e3b1004eb67cb651 0 1615365163512 3 connected2c06092fafa99e0158e39e6237a04fed25be3550 100.82.197.130:6379@16379 myself,master - 0 1615365162000 1 connected 0-54601379d2b20f26ab13d53068d276ec5d988b7a0273 100.64.122.197:6379@16379 slave 1895cb4b9c31b848666c61000e502f55a29a8255 0 1615365163000 2 connectede83446c368839c5fdccf5f70e3b1004eb67cb651 100.107.55.68:6379@16379 master - 0 1615365164014 3 connected 10923-16383

你的鼓励让我更有动力

赞赏

0人进行了赞赏支持

背景