Skip to main content

Upgrade via ocboot

This document describes how to upgrade service versions to a specified version using ocboot.

tip

ocboot can only upgrade environments deployed via ocboot. If your environment was deployed using docker compose or helm, please refer to the upgrade operations in the corresponding deployment documentation.

Prerequisites​

Version upgrades are recommended to be done incrementally between adjacent versions. For example, upgrading from v3.8.x to v3.10.x requires the following steps:

  1. v3.8.x => v3.9.x
  2. v3.9.x => v3.10.x

In general, the upgrade steps are as follows:

Use our ocboot tool for upgrades. This tool primarily calls ansible to upgrade all nodes in the cluster.

  1. Download the ocboot code for the corresponding version
  2. Use the ./ocboot.sh upgrade command in ocboot to perform the version upgrade

Check Current Version​

You can use kubectl to check the current cluster version

# 使用 kubectl 获得当前集群的版本
$ kubectl -n onecloud get onecloudclusters default -o=jsonpath='{.spec.version}'
v3.10.15 # 发现当前版本为 v3.10.15

Download ocboot Tool​

tip

If you already have the corresponding version code locally, you can skip the following steps. However, generally speaking, with each new version release, there will be a corresponding ocboot deployment tool release, and you need to use the corresponding version to update.

Download Deployment Tool​

The deployment tool code is at https://github.com/yunionio/ocboot/release, select the corresponding version, and download the tar.gz package of the code.

$ wget https://github.com/yunionio/ocboot/archive/refs/tags/master-v3.11.12-6.tar.gz
$ tar xf master-v3.11.12-6.tar.gz
$ cd ocboot-master-v3.11.12-6

Upgrade Service Components​

Prerequisites​

The principle of updating services is to remotely log in to the first control node of the cluster via SSH passwordless authentication from the local machine, obtain information about all nodes, and then execute playbook updates through ansible. Therefore, the following requirements must be met:

  1. The local machine can SSH remotely log in to PRIMARY_MASTER_HOST
  2. PRIMARY_MASTER_HOST can SSH passwordless log in to other nodes in the cluster

If passwordless login is not set up, please use the ssh-copy-id -i ~/.ssh/id_rsa.pub root@PRIMARY_MASTER_HOST command to distribute the public key to the corresponding nodes in your environment.

Upgradeable version numbers can be queried on the Release Notes 3.11 page.

Execute Upgrade Command​

The following command uses ocboot to upgrade related services to a higher version. This step will take a long time waiting for docker images to be pulled, please be patient.

PRIMARY_MASTER_HOST 是指部署集群的第一个节点的 ip 地址，需要本机能够使用 ssh 密钥登录上去。

$ ./ocboot.sh upgrade <PRIMARY_MASTER_HOST> v3.11.12

You can also use ./ocboot.sh upgrade --help to view other optional parameters.

  • --user can specify other SSH users
  • --port specifies the SSH port
  • --key-file specifies another SSH private key
  • --as-bastion can make PRIMARY_MASTER_HOST act as a bastion host to deploy hosts in the internal network

View Upgrade Process​

During the upgrade process, you can log in to PRIMARY_MASTER_HOST and use kubectl to view the upgrade status of corresponding pods.

$ kubectl get pods -n onecloud --watch

Downgrade​

If you encounter issues such as features not meeting expectations or bugs after upgrading, you can downgrade and rollback using the following commands.

warning
  • Generally, minor version downgrades are fine, such as downgrading from v3.10.7 to v3.10.6
  • Cross-version downgrades may have issues, such as downgrading from v3.10.7 to v3.9.15

If you encounter problems, please submit an issue on GitHub or contact us for help.

Rollback Operation​

Starting from ocboot v3.9.7, each time cluster services are upgraded, the operator's deployment and onecloudcluster's yaml files are backed up to /opt/yunion/ocboot/_upgrade/ for easy manual component rollback in the future.

When an upgrade fails, or you want to roll back to the version before the upgrade, you can use the following operations:

# View the upgrade backup directories in the /opt/yunion/ocboot/_upgrade/ directory
# Subdirectories are named with the pre-upgrade time
$ ls  /opt/yunion/ocboot/_upgrade/
2023.02.20-10:10:07

# For example, if you want to roll back to the version before 2023.02.20-10:10:07, perform the following operations
$ cd /opt/yunion/ocboot/_upgrade/2023.02.20-10\:10\:07/
# View related yaml files
$ ls -lh
-rw-r--r-- 1 root root  61K Feb 20 10:10 oc.yml # oc.yaml is the onecloudcluster description file
-rw-r--r-- 1 root root 3.9K Feb 20 10:10 operator.yml # operator.yaml is the operator deployment description file

# First roll back the operator
$ kubectl apply -f operator.yml
deployment.extensions/onecloud-operator configured
# Wait until the newly created operator becomes Running. If it's in ContainerCreating state, wait for a while
$ kubectl get pods -n onecloud | grep operator
onecloud-operator-5557d86d74-7s5sl                   1/1     Running   0          2s

# Then roll back the onecloudcluster
$ kubectl apply -f oc.yml