kubernetes存储之ceph-csi
16lz
2021-03-12
kubernetes存储之ceph-csi
scofield 菜鸟运维杂谈
0、前提
因为kubernetes默认的provisioner: kubernetes.io/rbd 无法正常使用，改用ceph官网提供的provisioner: rbd.csi.ceph.com，需要部署相关的csi插件以及配置ceph连接授权，所以本文记录如下。
1、下载部署清单
git clone https://github.com/ceph/ceph-csi.gitcd ceph-csi/deploy/rbd/kubernetes[root@qd01-stop-k8s-master001 kubernetes]# ls -ltotal 36-rw-r--r-- 1 root root 304 Feb 23 16:24 csi-config-map.yaml-rw-r--r-- 1 root root 1674 Feb 23 16:20 csi-nodeplugin-psp.yaml-rw-r--r-- 1 root root 747 Feb 23 16:20 csi-nodeplugin-rbac.yaml-rw-r--r-- 1 root root 1300 Feb 23 16:20 csi-provisioner-psp.yaml-rw-r--r-- 1 root root 2915 Feb 23 16:20 csi-provisioner-rbac.yaml-rw-r--r-- 1 root root 7123 Feb 23 16:34 csi-rbdplugin-provisioner.yaml-rw-r--r-- 1 root root 5841 Feb 23 16:34 csi-rbdplugin.yaml如下是需要的镜像
k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.2k8s.gcr.io/sig-storage/csi-attacher:v3.0.2k8s.gcr.io/sig-storage/csi-resizer:v1.0.1k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1quay.io/cephcsi/cephcsi:canary如果您的网络不能从k8s.gcr.io下载镜像，可以改成如下scofield/csi-provisioner:v2.0.4scofield/csi-snapshotter:v3.0.2scofield/csi-attacher:v3.0.2scofield/csi-resizer:v1.0.1scofield/csi-node-driver-registrar:v2.0.1scofield/cephcsi:canary2、修改配置
将kms所在行注释掉，因为没有这个配置，否则会部署失败
root@qd01-stop-k8s-master001 kubernetes]# vim csi-rbdplugin-provisioner.yaml[root@qd01-stop-k8s-master001 kubernetes]# vim csi-rbdplugin.yaml #- name: ceph-csi-encryption-kms-config # mountPath: /etc/ceph-csi-encryption-kms-config/ #- name: ceph-csi-encryption-kms-config # configMap: # name: ceph-csi-encryption-kms-config3、执行部署
[root@qd01-stop-k8s-master001 kubernetes]# kubectl apply -f . -n csiconfigmap/ceph-csi-config createdpodsecuritypolicy.policy/rbd-csi-nodeplugin-psp createdrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin-psp createdrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-psp createdserviceaccount/rbd-csi-nodeplugin createdclusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin createdpodsecuritypolicy.policy/rbd-csi-provisioner-psp createdrole.rbac.authorization.k8s.io/rbd-csi-provisioner-psp createdrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-psp createdserviceaccount/rbd-csi-provisioner createdclusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner createdclusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role createdrole.rbac.authorization.k8s.io/rbd-external-provisioner-cfg createdrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg createdservice/csi-rbdplugin-provisioner createddeployment.apps/csi-rbdplugin-provisioner createddaemonset.apps/csi-rbdplugin createdservice/csi-metrics-rbdplugin created等待部署完成即可[root@qd01-stop-k8s-master001 UseRBD]# kubectl get po -n csiNAME READY STATUS RESTARTS AGEcsi-rbdplugin-5xtbz 3/3 Running 0 29mcsi-rbdplugin-hwrsr 3/3 Running 0 29mcsi-rbdplugin-mtscj 3/3 Running 0 29mcsi-rbdplugin-pmqjv 3/3 Running 0 29mcsi-rbdplugin-provisioner-b96dc4989-fd7kt 7/7 Running 0 29mcsi-rbdplugin-provisioner-b96dc4989-tk9bv 7/7 Running 0 29mcsi-rbdplugin-provisioner-b96dc4989-xrxgz 7/7 Running 0 29mcsi-rbdplugin-qzsjr 3/3 Running 0 29mcsi-rbdplugin-tt4b9 3/3 Running 0 29mcsi-rbdplugin-w429q 3/3 Running 0 29mcsi-rbdplugin-w6xp7 3/3 Running 0 29mcsi-rbdplugin-wxc94 3/3 Running 0 29m4、使用ceph -rbd
1、创建需要的secret
创建csi-rbd-secret.yaml
---apiVersion: v1kind: Secretmetadata: name: csi-rbd-secret namespace: csistringData: userID: admin userKey: AQALpatf81ZmNhAAz6xt03v4boTYj7o5MOa0iQ==[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f csi-rbd-secret.yamlsecret/csi-rbd-secret created2、创建存储类sc
创建storageclass.yaml
---apiVersion: storage.k8s.io/v1kind: StorageClassmetadata: name: rbdprovisioner: rbd.csi.ceph.comparameters: clusterID: ec7ee19a-f7c6-4ed0-93a7-f48af473352c pool: k8s imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: csi csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: csi csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: csi csi.storage.k8s.io/fstype: xfsreclaimPolicy: DeleteallowVolumeExpansion: truemountOptions: - discard[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f storageclass.yamlstorageclass.storage.k8s.io/rbd created[root@qd01-stop-k8s-master001 UseRBD]# kubectl get scNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGErbd rbd.csi.ceph.com Delete Immediate true 20m3、创建PVC验证sc是否可用
创建raw-block-pvc.yaml
---apiVersion: v1kind: PersistentVolumeClaimmetadata: name: raw-block-pvcspec: accessModes: - ReadWriteOnce volumeMode: Block resources: requests: storage: 1Gi storageClassName: rbd[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f raw-block-pvc.yamlpersistentvolumeclaim/raw-block-pvc created[root@qd01-stop-k8s-master001 UseRBD]# kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEraw-block-pvc Bound pvc-84bf2ffb-7aee-41bd-9e6d-614c9f29eab4 1Gi RWO rbd 39s4、测试动态分配PVC
创建demo-statefulset-csi.yaml
apiVersion: apps/v1kind: StatefulSetmetadata: name: demo-nginx namespace: default labels: app: demo-nginxspec: serviceName: demo-nginx replicas: 2 selector: matchLabels: app: demo-nginx template: metadata: labels: app: demo-nginx spec: terminationGracePeriodSeconds: 180 initContainers: - name: init image: busybox command: ["chmod","777","-R","/data"] imagePullPolicy: Always volumeMounts: - name: volume mountPath: /data containers: - name: demo-nginx image: nginx ports: - containerPort: 80 name: port volumeMounts: - name: volume mountPath: /data volumeClaimTemplates: - metadata: name: volume spec: accessModes: ["ReadWriteOnce"] storageClassName: rbd resources: requests: storage: 5Gi[root@qd01-stop-k8s-master001 UseRBD]# kubectl apply -f demo-statefulset-csi.yamlstatefulset.apps/demo-nginx created5、验证可以看到,pvs自动创建好，并且正常挂载到pod中
[root@qd01-stop-k8s-master001 UseRBD]# kubectl get pvcNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEvolume-demo-nginx-0 Bound pvc-b0e3c919-10ad-49f7-a225-4337c07133ea 5Gi RWO rbd 6m37svolume-demo-nginx-1 Bound pvc-cb526baa-62ae-43ee-a544-eb1655c9c8c6 5Gi RWO rbd 2m24s[root@qd01-stop-k8s-master001 UseRBD]# kubectl get po NAME READY STATUS RESTARTS AGEdemo-nginx-0 1/1 Running 0 5m5sdemo-nginx-1 1/1 Running 0 2m31s进入到其中一个pod中可以看到，挂载了一个/dev/rbd2块存储，大小为制定的5G[root@qd01-stop-k8s-master001 UseRBD]# kubectl exec -ti demo-nginx-0 /bin/shkubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.# df -hFilesystem Size Used Avail Use% Mounted on/dev/rbd2 5.0G 38M 5.0G 1% /dataPS：文章会同步到dev.kubeops.net
©著作权归作者所有：来自51CTO博客作者mb5fdb131956bf3的原创作品，如需转载，请注明出处，否则将追究法律责任更多相关文章
- SQL基础知识V2——SQL语句快速参考
- ceph-块设备部署
- 了解pod和pod的生命周期-这一篇文章就够了
- SQL基础知识V2——索引
- Kubernetes中部署MySQL高可用集群
- 小技巧 | 如何在SQL Server中快速创建测试数据？
- k8s中蓝绿部署、金丝雀发布、滚动更新汇总
- 3.14 为vCenter Server服务器添加外部DSN连接
- 03-K8s部署安装配置nginx-ingress和配置外网访问