ExaCC: Upgrade Grid Infrastructure to 23ai using OCI CLI

In May 2024 Oracle announced the general availability of Oracle 23ai on ExaCC. In this blog post, I will upgrade an existing VM Cluster to 23ai using the OCI CLI.

If you want to use the OCI Web Console, please have a look at the blog post of Oracle ACE Director Fernando Simon: Upgrading to GI 23ai at ExaCC.

Requirements

Exadata Software Version

Before you can upgrade an existing VM Cluster the following minimum Exadata Software Versions need to be installed.

  • Exadata Infrastructure: 23.1.x or higher
  • Exadata Guest VM: 23.1.8 or higher

ASM RDBMS Compatibility Version

23ai RACs only support database version 19c and higher. Databases with older versions must be terminated before the upgrade. The ASM database compatibility version must be set to 19.0.0.0.0, otherwise, the precheck will raise the following error.

Disk group RDBMS compatibility setting ...FAILED
DATAC3: PRVE-3180 : RDBMS compatibility for ASM disk group "DATAC3" is set to
        "11.2.0.4.0", which is less than the minimum supported value
        "19.0.0.0.0".

RECOC3: PRVE-3180 : RDBMS compatibility for ASM disk group "RECOC3" is set to
        "11.2.0.4.0", which is less than the minimum supported value
        "19.0.0.0.0".

To change the ASM Diskgroup attribute, connect to the ASM instance with SYSASM privilege.

SQL> SELECT name, compatibility, database_compatibility FROM v$asm_diskgroup;

NAME       COMPATIBILITY        DATABASE_COMPATIBILI
---------- -------------------- --------------------
DATAC3     19.0.0.0.0           11.2.0.4.0
RECOC3     19.0.0.0.0           11.2.0.4.0

SQL> ALTER DISKGROUP DATAC3 SET ATTRIBUTE 'compatible.rdbms' = '19.0.0.0.0';
SQL> ALTER DISKGROUP RECOC3 SET ATTRIBUTE 'compatible.rdbms' = '19.0.0.0.0';

Precheck

Before we can run the precheck, we need the OCID of the Grid Infrastructure Upgrade. Use the following OCI CLI call to retrieve all available updates of the type GI_UPGRADE.

$> oci db vm-cluster list-updates \
   --vm-cluster-id ocid1.vmcluster.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
   --update-type GI_UPGRADE \
   --query "data[].{\"01 Name\" : \"description\", \"02 Version\" : \"version\", \"03 OCID\" : \"id\", \"04 Actions\" : \"available-actions\"}" \
   --output table
   
+---------------------------------------------------+--------------+------------------------------------------------------------------------------------------------+-------------------------------+------------+
| 01 Name                                           | 02 Version   | 03 OCID                                                                                        | 04 Actions                    | 05 Update  |
+---------------------------------------------------+--------------+------------------------------------------------------------------------------------------------+-------------------------------+------------+
| Grid Infrastructure Release Update : 23.4.0.24.05 | 23.4.0.24.05 | ocid1.dbupdate.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | ['ROLLING_APPLY', 'PRECHECK'] | GI_UPGRADE |
+---------------------------------------------------+--------------+------------------------------------------------------------------------------------------------+-------------------------------+------------+

In the next step run the precheck using the returned OCID.

$> oci db vm-cluster update \
   --vm-cluster-id ocid1.vmcluster.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
   --update-action PRECHECK \
   --update-id ocid1.dbupdate.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The easiest way to monitor the initiated work request is to log in to the OCI Web Console and navigate to the VM Cluster overview page.

Work Request for the Precheck

Upgrade

After the precheck is successful, the upgrade can be started with the following command.

$> oci db vm-cluster update \
   --vm-cluster-id ocid1.vmcluster.oc1.eu-frankfurt-1.abtheljrphmoulymvecagl6bf5gd2d7yaoc6kbqxrbpnfmgse7owjucwc6ba \
   --update-action ROLLING_APPLY \
   --update-id ocid1.dbupdate.oc1.eu-frankfurt-1.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Again, monitor the work request in the OCI Web Console.

Work Request for the Upgrade

Troubleshooting

Log Files

The log files of the work requests are created on the VM Cluster Nodes in the directory /var/opt/oracle/log/grid/upgrade.

$> ls -ltr /var/opt/oracle/log/grid/upgrade/
total 7192
-rw-r----- 1 oracle oinstall 1057448 Jun  4 14:41 pilot_2024-06-04_02-37-13-PM_10502
-rw-r----- 1 oracle oinstall   52634 Jun  4 14:41 dbaastools_2024-06-04_02-37-07-PM_10055.log
-rw-r----- 1 oracle oinstall 2785169 Jun  4 15:12 pilot_2024-06-04_02-43-54-PM_51097
-rw-r----- 1 oracle oinstall   70354 Jun  4 15:12 dbaastools_2024-06-04_02-43-49-PM_50728.log

Run CVU

To run the CVU we need to download the 23ai image. You can download the image with dbaascli. In the first step identify the available images.

$> dbaascli cswLib showImages --product GRID
DBAAS CLI version 24.1.2.0.0
Executing command cswLib showImages --product GRID
Job id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Session log: /var/opt/oracle/log/cswLib/showImages/dbaastools_2024-06-04_12-34-46-PM_306641.log
Log file location: /var/opt/oracle/log/cswLib/showImages/dbaastools_2024-06-04_12-34-46-PM_306641.log

############ List of Available grid Artifacts  #############

1.IMAGE_TAG=grid_19.22.0.0.0
  VERSION=19.22.0.0.0
  DESCRIPTION=19c JAN 2024 GI Image
2.IMAGE_TAG=grid_19.23.0.0.0
  VERSION=19.23.0.0.0
  DESCRIPTION=19c APR 2024 GI Image
3.IMAGE_TAG=grid_23.4.0.24.05
  VERSION=23.4.0.24.05
  DESCRIPTION=23ai grid image 23.4.0.24.05
4.IMAGE_TAG=grid_19.20.0.0.0
  VERSION=19.20.0.0.0
  DESCRIPTION=19c JUL 2023 GI Image
5.IMAGE_TAG=grid_19.21.0.0.0
  VERSION=19.21.0.0.0
  DESCRIPTION=19c OCT 2023 GI Image
Images can be downloaded using their image tags. For details, see help using 'dbaascli cswlib download --help'.
dbaascli execution completed

To download the image use the IMAGE_TAG.

$> dbaascli cswLib download --imageTag grid_23.4.0.24.05
DBAAS CLI version 24.1.2.0.0
Executing command cswLib download --imageTag grid_23.4.0.24.05
Job id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Session log: /var/opt/oracle/log/cswLib/download/dbaastools_2024-06-04_12-35-43-PM_309874.log
Image location=/var/opt/oracle/dbaas_acfs/gi_images/grid_23.4.0.24.05
Download succeeded
Log file location: /var/opt/oracle/log/cswLib/download/dbaastools_2024-06-04_12-35-43-PM_309874.log
dbaascli execution completed

$> ls -l /var/opt/oracle/dbaas_acfs/gi_images/grid_23.4.0.24.05
total 1021956
-rwx------ 1 grid oinstall 1046182666 Jun  4 09:17 grid_23.4.0.24.05_Image.zip

Now you can unzip the file and run the Cluster Verification Utility (CVU or cluvfy) to verify if your system is ready for the upgrade.

# Run as root user
$> mkdir /u02/grid
$> chown grid:oinstall /u02/grid
$> su - grid

# Run as grid user
$> unzip -q -d /u02/grid /var/opt/oracle/dbaas_acfs/gi_images/grid_23.4.0.24.05/grid_23.4.0.24.05_Image.zip
$> /u02/grid/runcluvfy.sh stage -pre crsinst -upgrade -src_crshome /u01/app/19.0.0.0/grid -dest_crshome /u02/grid -dest_version 23.0.0.0.0 -rolling

...
Pre-check for cluster services setup was successful.


Warnings were encountered during execution of CVU verification request "stage -pre crsinst".

RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.

Refer to My Oracle Support notes "2548970.1" for more details regarding errors
PRVG-11250".

Kernel crash dumping mechanism ...WARNING
oranode001: PRVE-10329 : Kernel crash dumping mechanism was not active on
                 node "oranode001". Command "/bin/systemctl is-active
                 kdump.service" returned "failed"

oranode002: PRVE-10329 : Kernel crash dumping mechanism was not active on
                 node "oranode002". Command "/bin/systemctl is-active
                 kdump.service" returned "failed"


CVU operation performed:      stage -pre crsinst
Date:                         Jun 4, 2024, 2:03:33 PM
CVU version:                  Standalone 23.4.0.24.5 (042324x8664)
Clusterware version:          19.0.0.0.0
CVU home:                     /u02/grid
Grid home:                    /u01/app/19.0.0.0/grid
User:                         grid
Operating system:             Linux5.4.17-2136.322.6.4.el8uek.x86_64

Changes

Oracle Home

The Oracle Home location changed from /u01/app/19.0.0.0/grid to /u02/app/23.0.0.0/gridhome_1. Oracle reduced the size of the Oracle Home drastically.

$> du -sh /u02/app/23.0.0.0/gridhome_1
3.1G    /u02/app/23.0.0.0/gridhome_1

The old Oracle Home /u01/app/19.0.0.0/grid still exists and will not be cleaned up during the upgrade.

$> du -sh /u01/app/19.0.0.0/grid
12G     /u01/app/19.0.0.0/grid

References

Leave a Reply

Your email address will not be published. Required fields are marked *