In May 2024 Oracle announced the general availability of Oracle 23ai on ExaCC. In this blog post, I will upgrade an existing VM Cluster to 23ai using the OCI CLI.
If you want to use the OCI Web Console, please have a look at the blog post of Oracle ACE Director Fernando Simon: Upgrading to GI 23ai at ExaCC.
Table of Contents
Requirements
Exadata Software Version
Before you can upgrade an existing VM Cluster the following minimum Exadata Software Versions need to be installed.
- Exadata Infrastructure: 23.1.x or higher
- Exadata Guest VM: 23.1.8 or higher
ASM RDBMS Compatibility Version
23ai RACs only support database version 19c and higher. Databases with older versions must be terminated before the upgrade. The ASM database compatibility version must be set to 19.0.0.0.0, otherwise, the precheck will raise the following error.
Disk group RDBMS compatibility setting ...FAILED
DATAC3: PRVE-3180 : RDBMS compatibility for ASM disk group "DATAC3" is set to
"11.2.0.4.0", which is less than the minimum supported value
"19.0.0.0.0".
RECOC3: PRVE-3180 : RDBMS compatibility for ASM disk group "RECOC3" is set to
"11.2.0.4.0", which is less than the minimum supported value
"19.0.0.0.0".
To change the ASM Diskgroup attribute, connect to the ASM instance with SYSASM privilege.
SQL> SELECT name, compatibility, database_compatibility FROM v$asm_diskgroup;
NAME COMPATIBILITY DATABASE_COMPATIBILI
---------- -------------------- --------------------
DATAC3 19.0.0.0.0 11.2.0.4.0
RECOC3 19.0.0.0.0 11.2.0.4.0
SQL> ALTER DISKGROUP DATAC3 SET ATTRIBUTE 'compatible.rdbms' = '19.0.0.0.0';
SQL> ALTER DISKGROUP RECOC3 SET ATTRIBUTE 'compatible.rdbms' = '19.0.0.0.0';
Precheck
Before we can run the precheck, we need the OCID of the Grid Infrastructure Upgrade. Use the following OCI CLI call to retrieve all available updates of the type GI_UPGRADE.
$> oci db vm-cluster list-updates \
--vm-cluster-id ocid1.vmcluster.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
--update-type GI_UPGRADE \
--query "data[].{\"01 Name\" : \"description\", \"02 Version\" : \"version\", \"03 OCID\" : \"id\", \"04 Actions\" : \"available-actions\"}" \
--output table
+---------------------------------------------------+--------------+------------------------------------------------------------------------------------------------+-------------------------------+------------+
| 01 Name | 02 Version | 03 OCID | 04 Actions | 05 Update |
+---------------------------------------------------+--------------+------------------------------------------------------------------------------------------------+-------------------------------+------------+
| Grid Infrastructure Release Update : 23.4.0.24.05 | 23.4.0.24.05 | ocid1.dbupdate.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | ['ROLLING_APPLY', 'PRECHECK'] | GI_UPGRADE |
+---------------------------------------------------+--------------+------------------------------------------------------------------------------------------------+-------------------------------+------------+
In the next step run the precheck using the returned OCID.
$> oci db vm-cluster update \
--vm-cluster-id ocid1.vmcluster.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
--update-action PRECHECK \
--update-id ocid1.dbupdate.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
The easiest way to monitor the initiated work request is to log in to the OCI Web Console and navigate to the VM Cluster overview page.
Upgrade
After the precheck is successful, the upgrade can be started with the following command.
$> oci db vm-cluster update \
--vm-cluster-id ocid1.vmcluster.oc1.eu-frankfurt-1.abtheljrphmoulymvecagl6bf5gd2d7yaoc6kbqxrbpnfmgse7owjucwc6ba \
--update-action ROLLING_APPLY \
--update-id ocid1.dbupdate.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Again, monitor the work request in the OCI Web Console.
Troubleshooting
Log Files
The log files of the work requests are created on the VM Cluster Nodes in the directory /var/opt/oracle/log/grid/upgrade.
$> ls -ltr /var/opt/oracle/log/grid/upgrade/
total 7192
-rw-r----- 1 oracle oinstall 1057448 Jun 4 14:41 pilot_2024-06-04_02-37-13-PM_10502
-rw-r----- 1 oracle oinstall 52634 Jun 4 14:41 dbaastools_2024-06-04_02-37-07-PM_10055.log
-rw-r----- 1 oracle oinstall 2785169 Jun 4 15:12 pilot_2024-06-04_02-43-54-PM_51097
-rw-r----- 1 oracle oinstall 70354 Jun 4 15:12 dbaastools_2024-06-04_02-43-49-PM_50728.log
Run CVU
To run the CVU we need to download the 23ai image. You can download the image with dbaascli. In the first step identify the available images.
$> dbaascli cswLib showImages --product GRID
DBAAS CLI version 24.1.2.0.0
Executing command cswLib showImages --product GRID
Job id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Session log: /var/opt/oracle/log/cswLib/showImages/dbaastools_2024-06-04_12-34-46-PM_306641.log
Log file location: /var/opt/oracle/log/cswLib/showImages/dbaastools_2024-06-04_12-34-46-PM_306641.log
############ List of Available grid Artifacts #############
1.IMAGE_TAG=grid_19.22.0.0.0
VERSION=19.22.0.0.0
DESCRIPTION=19c JAN 2024 GI Image
2.IMAGE_TAG=grid_19.23.0.0.0
VERSION=19.23.0.0.0
DESCRIPTION=19c APR 2024 GI Image
3.IMAGE_TAG=grid_23.4.0.24.05
VERSION=23.4.0.24.05
DESCRIPTION=23ai grid image 23.4.0.24.05
4.IMAGE_TAG=grid_19.20.0.0.0
VERSION=19.20.0.0.0
DESCRIPTION=19c JUL 2023 GI Image
5.IMAGE_TAG=grid_19.21.0.0.0
VERSION=19.21.0.0.0
DESCRIPTION=19c OCT 2023 GI Image
Images can be downloaded using their image tags. For details, see help using 'dbaascli cswlib download --help'.
dbaascli execution completed
To download the image use the IMAGE_TAG.
$> dbaascli cswLib download --imageTag grid_23.4.0.24.05
DBAAS CLI version 24.1.2.0.0
Executing command cswLib download --imageTag grid_23.4.0.24.05
Job id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Session log: /var/opt/oracle/log/cswLib/download/dbaastools_2024-06-04_12-35-43-PM_309874.log
Image location=/var/opt/oracle/dbaas_acfs/gi_images/grid_23.4.0.24.05
Download succeeded
Log file location: /var/opt/oracle/log/cswLib/download/dbaastools_2024-06-04_12-35-43-PM_309874.log
dbaascli execution completed
$> ls -l /var/opt/oracle/dbaas_acfs/gi_images/grid_23.4.0.24.05
total 1021956
-rwx------ 1 grid oinstall 1046182666 Jun 4 09:17 grid_23.4.0.24.05_Image.zip
Now you can unzip the file and run the Cluster Verification Utility (CVU or cluvfy) to verify if your system is ready for the upgrade.
# Run as root user
$> mkdir /u02/grid
$> chown grid:oinstall /u02/grid
$> su - grid
# Run as grid user
$> unzip -q -d /u02/grid /var/opt/oracle/dbaas_acfs/gi_images/grid_23.4.0.24.05/grid_23.4.0.24.05_Image.zip
$> /u02/grid/runcluvfy.sh stage -pre crsinst -upgrade -src_crshome /u01/app/19.0.0.0/grid -dest_crshome /u02/grid -dest_version 23.0.0.0.0 -rolling
...
Pre-check for cluster services setup was successful.
Warnings were encountered during execution of CVU verification request "stage -pre crsinst".
RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.
Refer to My Oracle Support notes "2548970.1" for more details regarding errors
PRVG-11250".
Kernel crash dumping mechanism ...WARNING
oranode001: PRVE-10329 : Kernel crash dumping mechanism was not active on
node "oranode001". Command "/bin/systemctl is-active
kdump.service" returned "failed"
oranode002: PRVE-10329 : Kernel crash dumping mechanism was not active on
node "oranode002". Command "/bin/systemctl is-active
kdump.service" returned "failed"
CVU operation performed: stage -pre crsinst
Date: Jun 4, 2024, 2:03:33 PM
CVU version: Standalone 23.4.0.24.5 (042324x8664)
Clusterware version: 19.0.0.0.0
CVU home: /u02/grid
Grid home: /u01/app/19.0.0.0/grid
User: grid
Operating system: Linux5.4.17-2136.322.6.4.el8uek.x86_64
Changes
Oracle Home
The Oracle Home location changed from /u01/app/19.0.0.0/grid to /u02/app/23.0.0.0/gridhome_1. Oracle reduced the size of the Oracle Home drastically.
$> du -sh /u02/app/23.0.0.0/gridhome_1
3.1G /u02/app/23.0.0.0/gridhome_1
The old Oracle Home /u01/app/19.0.0.0/grid still exists and will not be cleaned up during the upgrade.
$> du -sh /u01/app/19.0.0.0/grid
12G /u01/app/19.0.0.0/grid
One comment