控制节点替换流程
背景
即使是高可用部署的 3 节点 cloudpods over k8s 集群，在生产环境可能会出现任意 1 个节点挂掉的情况。 一些普通的故障，比如更换内存，CPU 之类可以解决的问题，可以临时关机，恢复后再重启节点解决。
但如果发生硬盘之类的故障，比如数据无法恢复的情况，需要把该节点删除再重新加入新的节点，接下来描述该步骤以及注意事项。
测试环境
- k8s_vip: 10.127.100.102
- 使用 staticpods keepalived 运行在 3 个 master 节点上
- 由 kubelet 直接启动，路径：/etc/kubernetes/manifests/keepalived.yaml
- primary_master_node 第1个初始化的控制节点: ip: 10.127.100.234
- master_node_1 第2个加入控制节点: ip: 10.127.100.229
- master_node_2 第3个加入的控制节点: ip: 10.127.100.226
- 数据库: 数据库部署在集群之外，不在 3 个节点之上
- CSI: 使用 local-path
- local-path 的 CSI 会强绑定 pod 到指定的 node，这里需要特别注意
- 如果挂掉的节点有对应 local-path 的 pvc 绑定在上面，使用该 pvc 的 pod 就没办法漂移到其它 Ready 的 node ，这种 pod 叫有状态的 pod，可以通过命令
kubectl get pvc -A | grep local-path就可以看到所有的 local-path pvc
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
lzx-ocboot-ha-test Ready master 100m v1.15.12 10.127.100.234 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
lzx-ocboot-ha-test-2 Ready master 61m v1.15.12 10.127.100.229 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
lzx-ocboot-ha-test-3 Ready master 60m v1.15.12 10.127.100.226 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
- minio:
- 高可用部署下使用 minio 作为 glance 的后端存储
- statefulset
- 使用 local-path CSI 作为后端存储
$ kubectl get pods -n onecloud-minio -o wide
kubectl get pods -n onecloud-minio -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
minio-0 1/1 Running 0 46m 10.40.99.205 lzx-ocboot-ha-test <none> <none>
minio-1 1/1 Running 0 46m 10.40.158.215 lzx-ocboot-ha-test-3 <none> <none>
minio-2 1/1 Running 0 46m 10.40.159.22 lzx-ocboot-ha-test-2 <none> <none>
minio-3 1/1 Running 0 46m 10.40.99.206 lzx-ocboot-ha-test <none> <none>
$ kubectl get pvc -n onecloud-minio
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
export-minio-0 Bound pvc-297ed5e5-66c8-4855-8031-c65a0ccfa4d0 1Ti RWO local-path 46m
export-minio-1 Bound pvc-4e8fe486-5b23-44a0-876c-df36d134957f 1Ti RWO local-path 46m
export-minio-2 Bound pvc-389b3c61-6000-4757-9949-db53e4e53776 1Ti RWO local-path 46m
export-minio-3 Bound pvc-3dd54509-7745-47dd-84ea-fbacfe1e2f5b 1Ti RWO local-path 46m
测试
目标
下线 primary_master_node 10.127.100.234 节点，加入新的节点替换该节点
步骤
1. 需要确认有哪些有状态的 pod 和 pvc 运行在该节点
# 根据 IP 找到节点在 k8s 集群中的名称
$ kubectl get nodes -o wide | grep 10.127.100.234
lzx-ocboot-ha-test Ready master 4h15m v1.15.12 10.127.100.234 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
# 查看所有为 local-path 的 pvc
$ kubectl get pvc -A | grep local-path
# onecloud-minio namespace 的 export-minio-x 负责存储 glance 的镜像，是关键组件
onecloud-minio export-minio-0 Bound pvc-297ed5e5-66c8-4855-8031-c65a0ccfa4d0 1Ti RWO local-path 3h11m
onecloud-minio export-minio-1 Bound pvc-4e8fe486-5b23-44a0-876c-df36d134957f 1Ti RWO local-path 3h11m
onecloud-minio export-minio-2 Bound pvc-389b3c61-6000-4757-9949-db53e4e53776 1Ti RWO local-path 3h11m
onecloud-minio export-minio-3 Bound pvc-3dd54509-7745-47dd-84ea-fbacfe1e2f5b 1Ti RWO local-path 3h11m
# onecloud-monitoring namespace 的 export-monitor-minio-x 负责存储服务的日志，不是关键组件
onecloud-monitoring export-monitor-minio-0 Bound pvc-b885605f-b5ca-40ff-b968-4d95b03e8bb8 1Ti RWO local-path 3h8m
onecloud-monitoring export-monitor-minio-1 Bound pvc-520a8262-5dad-48aa-9a0e-0e25f850faad 1Ti RWO local-path 3h8m
onecloud-monitoring export-monitor-minio-2 Bound pvc-6de1ff0f-3465-4a51-8124-880f1b3c6d7a 1Ti RWO local-path 3h8m
onecloud-monitoring export-monitor-minio-3 Bound pvc-364652ca-496e-4b29-82ea-ec7e768aa8f5 1Ti RWO local-path 3h8m
# onecloud namespace 下面的这些 pvc 都是系统服务依赖的
# default-baremetal-agent 是存储物理机管理的存储，不开启 baremetal-agent 可以不用管，默认也是 Pending 待绑定状态
onecloud default-baremetal-agent Pending local-path 3h35m
# default-esxi-agent 负责存 esxi-agent 服务相关的本地数据
onecloud default-esxi-agent Bound pvc-b32adfcc-96e7-45e4-b8bd-b5318c954dca 30G RWO local-path 3h35m
# default-glance 负责存 glance 服务的镜像，在高可用部署环境下，deployment default-glance 不会挂载这个 pvc ，会使用 onecloud-minio 里面的 minio s3 存储镜像，可以不用管
onecloud default-glance Bound pvc-e6ee398e-2d84-46cf-9401-e94f438d87cd 100G RWO local-path 3h36m
# default-influxdb 负责存平台的监控数据，监控数据可以容忍丢失，如果所在节点挂了，可以删掉重建
onecloud default-influxdb Bound pvc-871b9441-c56f-4bb4-8b56-868e1df1a438 20G RWO local-path 3h35m
# 查看该节点上有哪些 pod
$ kubectl get pods -A -o wide | grep onecloud | grep 'lzx-ocboot-ha-test '
onecloud-minio minio-0 1/1 Running 0 3h10m 10.40.99.205 lzx-ocboot-ha-test <none> <none>
onecloud-minio minio-3 1/1 Running 0 3h10m 10.40.99.206 lzx-ocboot-ha-test <none> <none>
onecloud-monitoring monitor-kube-state-metrics-6c97499758-w69tz 1/1 Running 0 3h6m 10.40.99.214 lzx-ocboot-ha-test <none> <none>
onecloud-monitoring monitor-loki-0 1/1 Running 0 3h6m 10.40.99.213 lzx-ocboot-ha-test <none> <none>
onecloud-monitoring monitor-minio-0 1/1 Running 0 3h7m 10.40.99.211 lzx-ocboot-ha-test <none> <none>
onecloud-monitoring monitor-minio-3 1/1 Running 0 3h7m 10.40.99.212 lzx-ocboot-ha-test <none> <none>
onecloud-monitoring monitor-monitor-stack-operator-54d8c46577-qknws 1/1 Running 0 3h6m 10.40.99.216 lzx-ocboot-ha-test <none> <none>
onecloud-monitoring monitor-promtail-4mx2s 1/1 Running 0 3h6m 10.40.99.215 lzx-ocboot-ha-test <none> <none>
onecloud default-etcd-7brtldv78z 1/1 Running 0 3h10m 10.40.99.207 lzx-ocboot-ha-test <none> <none>
onecloud default-glance-6fd697b7b9-nbk9t 1/1 Running 0 3h7m 10.40.99.208 lzx-ocboot-ha-test <none> <none>
onecloud default-host-5rmg8 3/3 Running 7 3h34m 10.127.100.234 lzx-ocboot-ha-test <none> <none>
onecloud default-host-deployer-sf494 1/1 Running 7 3h34m 10.40.99.202 lzx-ocboot-ha-test <none> <none>
onecloud default-host-image-s6pwq 1/1 Running 2 3h34m 10.127.100.234 lzx-ocboot-ha-test <none> <none>
onecloud default-region-dns-2hcpv 1/1 Running 1 3h34m 10.127.100.234 lzx-ocboot-ha-test <none> <none>
onecloud default-telegraf-5jn4x 2/2 Running 0 3h34m 10.127.100.234 lzx-ocboot-ha-test <none> <none>
onecloud default-influxdb-6bqgq 1/1 Running 0 3h34m 10.127.99.218 lzx-ocboot-ha-test <none> <none>
通过上面命令的结果，可以筛选出 onecloud-minio/minio-0 onecloud-minio/minio-3，onecloud/default-influxdb，onecloud-monitoring/monitor-minio-0,onecloud-minio/monitor-minio-3 这些有状态的 pod 在 primary_master_node 上。
2. 接下来将 primary_master_node 关机踢出集群
# 登录其它两个 master_node 节点，比如：10.127.100.229
$ ssh root@10.127.100.229
# 设置 KUBECONFIG 配置
[root@lzx-ocboot-ha-test-2 ~]$ export KUBECONFIG=/etc/kubernetes/admin.conf
# 查看节点状态，发现 primary_master_node 已经变成 NotReady
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
lzx-ocboot-ha-test NotReady master 4h37m v1.15.12
lzx-ocboot-ha-test-2 Ready master 3h58m v1.15.12
lzx-ocboot-ha-test-3 Ready master 3h57m v1.15.12
# 删除 primary_master_node 节点：lzx-ocboot-ha-test
[root@lzx-ocboot-ha-test-2 ~]$ kubectl drain --delete-local-data --ignore-daemonsets lzx-ocboot-ha-test
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-fdzql, kube-system/kube-proxy-nfxvd, kube-system/traefik-ingress-controller-jms9v, onecloud-monitoring/monitor-promtail-4mx2s, onecloud/default-host-5rmg8, onecloud/default-host-deployer-sf494, onecloud/default-host-image-s6pwq, onecloud/default-region-dns-2hcpv, onecloud/default-telegraf-5jn4x
evicting pod "minio-0"
evicting pod "monitor-minio-3"
evicting pod "default-etcd-7brtldv78z"
evicting pod "monitor-kube-state-metrics-6c97499758-w69tz"
evicting pod "default-influxdb-85945647d5-6bqgq"
evicting pod "default-glance-6fd697b7b9-nbk9t"
evicting pod "minio-3"
evicting pod "monitor-monitor-stack-operator-54d8c46577-qknws"
evicting pod "monitor-loki-0"
evicting pod "monitor-minio-0"
# 该命令会卡停住，因为 primary_master_node 已经关机了，无法删除 pod ，这个时候 'Ctrl-c' 取消命令
^C
# 使用 kubectl delete node 直接删除 primary_master_node 节点
$ kubectl delete node lzx-ocboot-ha-test
# 然后再查看处于 Pending 状态的 pod 全部都是之前 primary_master_node 上面有状态的 pod
# 因为这些 pod 使用了 local-path 的 pvc ，这些 pvc 是和节点强绑定的，还存在集群中
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get pods -A | grep Pending
onecloud-minio minio-0 0/1 Pending 0 61s
onecloud-minio minio-3 0/1 Pending 0 61s
onecloud-monitoring monitor-minio-0 0/1 Pending 0 61s
onecloud-monitoring monitor-minio-3 0/1 Pending 0 61s
onecloud default-influxdb-85945647d5-x5sv5 0/1 Pending 0 10m
3. 删除旧的 primary_master_node 的 etcd endpoint
# 查看 kube-system 系统下的 etcd pod
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get pods -n kube-system | grep etcd
etcd-lzx-ocboot-ha-test-2 1/1 Running 1 4h52m
etcd-lzx-ocboot-ha-test-3 1/1 Running 1 4h51m
# 进入 etcd-lzx-ocboot-ha-test-2 etcd pod
[root@lzx-ocboot-ha-test-2 ~]# kubectl exec -ti -n kube-system etcd-lzx-ocboot-ha-test-2 sh
# 使用 etcdctl 查看 member list
$ etcdctl --endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
# 会发现旧的 primary_master_node member lzx-ocboot-ha-test 还在 etcd 集群
14da7b338b44eee0, started, lzx-ocboot-ha-test, https://10.127.100.234:2380, https://10.127.100.234:2379, false
454ae6f931376261, started, lzx-ocboot-ha-test-2, https://10.127.100.229:2380, https://10.127.100.229:2379, false
5afd19948b9009f6, started, lzx-ocboot-ha-test-3, https://10.127.100.226:2380, https://10.127.100.226:2379, false
# 删除 lzx-ocboot-ha-test member
$ etcdctl --endpoints https://127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 14da7b338b44eee0
4. 替换旧的 primary_master_node 节点，加入新的 master_node
旧的 primary_master_node 节点已经被删掉，会发现 keepalived 的 vip 也漂移到了 master_node_1 上：
# 查看 vip 为 10.127.100.102
# 如果该节点运行了云平台 host 服务，ip 会绑定到 br0 上
[root@lzx-ocboot-ha-test-2 ~]$ ip addr show br0
32: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 00:22:96:6f:6e:f1 brd ff:ff:ff:ff:ff:ff
inet 10.127.100.229/24 brd 10.127.100.255 scope global br0
valid_lft forever preferred_lft forever
inet 10.127.100.102/32 scope global br0
valid_lft forever preferred_lft forever
inet6 fe80::222:96ff:fe6f:6ef1/64 scope link
valid_lft forever preferred_lft forever
# 查看 /etc/kubernetes/manifests/keepalived.yaml env 配置
[root@lzx-ocboot-ha-test-2 ~]$ cat /etc/kubernetes/manifests/keepalived.yaml | | grep -A 15 env
env:
# 对应 keepalived 的权重，除了 primary_master_node 的 keepalived 会设置 100，其余 master_node 上面的都是 90
- name: KEEPALIVED_PRIORITY
value: "90"
# 设置的 VIP
- name: KEEPALIVED_VIRTUAL_IPS
value: '#PYTHON2BASH:[''10.127.100.102'']'
# 是 BACKUP 角色
- name: KEEPALIVED_STATE
value: BACKUP
# 密码
- name: KEEPALIVED_PASSWORD
value: de17f785
# router id
- name: KEEPALIVED_ROUTER_ID
value: "12"
# 改节点网卡实际 ip
- name: KEEPALIVED_NODE_IP
value: 10.127.100.229
# keepalived 绑定的网卡
- name: KEEPALIVED_INTERFACE
value: eth0
image: registry.cn-beijing.aliyuncs.com/yunionio/keepalived:v2.0.25
# 查看集群当前只有 2 个节点
[root@lzx-ocboot-ha-test-2 ocboot]$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
lzx-ocboot-ha-test-2 Ready master 4h29m v1.15.12 10.127.100.229 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
lzx-ocboot-ha-test-3 Ready master 4h28m v1.15.12 10.127.100.226 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
现在使用 ocboot 加入新的节点，编辑 ocboot yaml 配置：
- 把当前的 lzx-ocboot-ha-test-2 这个节点当做 primary_master_node
- 需要新加入 master_node 节点信息：
- IP: 10.127.100.224
- Name: lzx-ocboot-ha-test-4
因为旧的 primary_master_node 已经被删除了，又把当前的 master_node_1 当做新集群的 primary_master_node，现在的配置变成下面这样:
$ cat config-new-k8s-ha.yaml
primary_master_node:
# 这里把之前的 master_node_1 10.127.100.229 当做 primary_master_node
hostname: 10.127.100.229
use_local: false
user: root
onecloud_version: "v3.8.8"
# 数据库连接信息，根据自己的环境填写
db_host: 10.127.100.101
db_user: "root"
db_password: "0neC1oudDB#"
db_port: "3306"
image_repository: registry.cn-beijing.aliyuncs.com/yunionio
ha_using_local_registry: false
node_ip: "10.127.100.229"
# keepalived 暴露出去的 vip
controlplane_host: 10.127.100.102
controlplane_port: "6443"
as_host: true
# 启用 ha ，默认部署 keepavlied
high_availability: true
use_ee: false
# 高可用使用 minio
enable_minio: true
host_networks: "eth0/br0/10.127.100.229"
master_nodes:
# 加入到 10.127.100.102 vip 的 k8s 集群
controlplane_host: 10.127.100.102
controlplane_port: "6443"
# 运行云平台控制相关组件
as_controller: true
# 作为云平台私有云宿主机计算节点
as_host: true
# 启用 keepavlied
high_availability: true
hosts:
- user: root
hostname: "10.127.100.224"
host_networks: "eth0/br0/10.127.100.224"
写好配置后，使用 ocboot 加入新节点。
下载 ocboot 部署工具代码。
# Use git clone the ocboot deployment tool locally
$ git clone -b release/3.10 https://github.com/yunionio/ocboot && cd ./ocboot
然后加入节点。
$ ./run.py config-new-k8s-ha.yaml
等到 ocboot 的 ./run.py 运行完成后，再查看 node 信息，发现新的节点 lzx-ocboot-ha-test-4(10.127.100.224) 已经加入进来：
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
lzx-ocboot-ha-test-2 Ready master 5h13m v1.15.12 10.127.100.229 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
lzx-ocboot-ha-test-3 Ready master 5h12m v1.15.12 10.127.100.226 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
lzx-ocboot-ha-test-4 Ready master 10m v1.15.12 10.127.100.224 <none> CentOS Linux 7 (Core) 3.10.0-1160.6.1.el7.yn20201125.x86_64 docker://20.10.5
修改新的 primary_master_node keepalived 的权重和角色
[root@lzx-ocboot-ha-test-2 ~]$ vim /etc/kubernetes/manifests/keepalived.yaml
...
# 把权重改成 100
- name: KEEPALIVED_PRIORITY
value: "100"
...
# 把角色改成 MASTER
- name: KEEPALIVED_STATE
value: MASTER
5. 恢复有状态的 pod
新的 master 节点加入后，会发现原来的有状态 pod 还是 Pending，接下来就需要删除旧的 pvc ，把它们启动到新的 master 节点上。
# 查看 Pending 状态的 pod
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get pods -A | grep Pending
onecloud-minio minio-0 0/1 Pending 0 74m
onecloud-minio minio-3 0/1 Pending 0 74m
onecloud-monitoring monitor-minio-0 0/1 Pending 0 74m
onecloud-monitoring monitor-minio-3 0/1 Pending 0 74m
onecloud default-influxdb-85945647d5-x5sv5 0/1 Pending 0 84m
# 把现在的 primary_master_node 和旧的 master_node 先禁止调度，这样可以保证后面的有状态 pod 创建到新的 master 节点上
# 保证 minio 多副本打散到不同的 master 节点
[root@lzx-ocboot-ha-test-2 ~]$ kubectl cordon lzx-ocboot-ha-test-2 lzx-ocboot-ha-test-3
首先恢复 onecloud-minio 里面的 minio statefulset 组件，因为里面存储了 glance 依赖的镜像，通过之前的命令发现 onecloud-minio namespace 里面的 minio-0 和 minio-3 是 Pending 状态，接下来删除它们依赖的 pvc ，然后 pod 就会重启到新的 master_node 节点上，操作如下：
# 找到对应的 pvc
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get pvc -n onecloud-minio | egrep 'minio-0|minio-3'
export-minio-0 Bound pvc-297ed5e5-66c8-4855-8031-c65a0ccfa4d0 1Ti RWO local-path 4h55m
export-minio-3 Bound pvc-3dd54509-7745-47dd-84ea-fbacfe1e2f5b 1Ti RWO local-path 4h55m
# 删除 pvc
[root@lzx-ocboot-ha-test-2 ~]$ kubectl delete pvc -n onecloud-minio export-minio-0 export-minio-3
# 删除 pod
[root@lzx-ocboot-ha-test-2 ~]$ kubectl delete pods -n onecloud-minio minio-0 minio-3
# 查看新创建启动 minio-0 和 minio-3 ，已经启动到新的 node lzx-ocboot-ha-test-4 上
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get pods -n onecloud-minio -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
minio-0 1/1 Running 0 7s 10.40.103.200 lzx-ocboot-ha-test-4 <none> <none>
minio-1 1/1 Running 0 5h1m 10.40.158.215 lzx-ocboot-ha-test-3 <none> <none>
minio-2 1/1 Running 0 5h1m 10.40.159.22 lzx-ocboot-ha-test-2 <none> <none>
minio-3 1/1 Running 0 14s 10.40.103.199 lzx-ocboot-ha-test-4 <none> <none>
# 查看 minio-3 的日志，发现已经自愈完毕
[root@lzx-ocboot-ha-test-2 ~]$ kubectl logs -n onecloud-minio minio-3
....
Healing disk '/export' on 1st pool
Healing disk '/export' on 1st pool complete
Summary:
{
"ID": "0e5c1947-44f0-4f8a-b7f0-e3a55f441d6f",
"PoolIndex": 0,
"SetIndex": 0,
"DiskIndex": 3,
"Path": "/export",
"Endpoint": "http://minio-3.minio-svc.onecloud-minio.svc.cluster.local:9000/export",
"Started": "2022-04-13T06:49:29.882069559Z",
"LastUpdate": "2022-04-13T06:49:59.564158167Z",
"ObjectsHealed": 10,
"ObjectsFailed": 0,
"BytesDone": 1756978429,
"BytesFailed": 0,
"QueuedBuckets": [],
"HealedBuckets": [
".minio.sys/config",
".minio.sys/buckets",
"onecloud-images"
]
}
...
# 也可以登录到 lzx-ocboot-ha-test-4 上，查看 local-path csi 里面的 minio onecloud-images bucket 里面有没有对应的镜像 parts
$ ssh root@10.127.100.224
# 进入对应的 pvc 目录，目录名可以使用 kubectl get pvc -n onecloud-minio | grep minio-3 获得
[root@lzx-ocboot-ha-test-4 ~]$ cd /opt/local-path-provisioner/pvc-352277cb-e69d-41bf-b58a-d65cb1e4e6f8/
[root@lzx-ocboot-ha-test-4 pvc-352277cb-e69d-41bf-b58a-d65cb1e4e6f8]$ du -smh onecloud-images/
838M onecloud-images/
然后使用恢复 onecloud-minio 相同的方法，恢复 monitor-minio 就可以，参考命令如下：
[root@lzx-ocboot-ha-test-2 ~]$ kubectl delete pvc -n onecloud-monitoring export-monitor-minio-0 export-monitor-minio-3
persistentvolumeclaim "export-monitor-minio-0" deleted
persistentvolumeclaim "export-monitor-minio-3" deleted
[root@lzx-ocboot-ha-test-2 ~]$ kubectl delete pods -n onecloud-monitoring monitor-minio-0 monitor-minio-3
pod "monitor-minio-0" deleted
pod "monitor-minio-3" deleted
[root@lzx-ocboot-ha-test-2 ~]$ kubectl get pods -n onecloud-monitoring | grep minio
monitor-minio-0 1/1 Running 0 24s
monitor-minio-1 1/1 Running 0 5h17m
monitor-minio-2 1/1 Running 0 5h17m
monitor-minio-3 1/1 Running 0 24s
恢复 influxdb deployment，influxdb 和 minio 不太一样，minio 使用的 statefulset 管理，删掉 pod 和 pvc 后，k8s 会自动创建对应序号的 pod 和 pvc ，但是 deployment 不会，所以恢复 influxdb 的步骤是删掉 pvc，然后同时删掉 default-influxdb 这个 deployment ，我们的 onecloud-operator 组件就会新建对应的资源，步骤如下：
# 恢复 influxdb
[root@lzx-ocboot-ha-test-2 ~]# kubectl delete pvc -n onecloud default-influxdb
[root@lzx-ocboot-ha-test-2 ~]# kubectl delete deployment -n onecloud default-influxdb
[root@lzx-ocboot-ha-test-2 ~]# kubectl get pods -n onecloud | grep influxdb
default-influxdb-85945647d5-mdd2z 1/1 Running 0 7m44s
现在所有在旧的 primary_master_node 上的组件都恢复，接下来把禁止调度的节点启用调度：
[root@lzx-ocboot-ha-test-2 ~]$ kubectl uncordon lzx-ocboot-ha-test-2 lzx-ocboot-ha-test-3