This blog post describes an issue with patchmgr while checking the SSH equivalency of the target cluster nodes. More details about patchmgr and how to use it to patch your VM clusters can be found here.
Table of Contents
Error
Before I update the operating system of my VM clusters, I ran the following precheck command.
$> /stage/dbserver_patch_21.211018/patchmgr -dbnodes /stage/nodes.conf \
-precheck \
-iso_repo "/stage/20.1.15.0.0.211013/exadata_ol7_20.1.15.0.0.211013_Linux-x86-64.zip" \
-target_version "20.1.15.0.0.211013" \
-allow_active_network_mounts \
-nomodify_at_prereq \
-log_dir /stage/logs
One of the first step is to check the SSH equivalency of all involded nodes – I am using a different server to initiate the patch procedure. This time patchmgr failed to connect to the nodes.
************************************************************************************************************
NOTE patchmgr release: 21.211018 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.
************************************************************************************************************
2021-11-19 10:06:20 +0100 :INFO : Checking hosts connectivity via ICMP/ping
2021-11-19 10:06:21 +0100 :INFO : Hosts Reachable: [node1 node2]
2021-11-19 10:06:21 +0100 :INFO : All hosts are reachable via ping/ICMP
2021-11-19 10:06:21 +0100 :Working: Verify SSH equivalence for the root user to node(s)
2021-11-19 10:06:21 +0100 :WARNING: patchmgr is launched from a remote/shared filesystem
2021-11-19 10:06:21 +0100 :WARNING: Ensure the filesystem is not hosted from any of the target nodes listed
2021-11-19 10:06:21 +0100 :ERROR : Cannot connect to node1 with SSH
2021-11-19 10:06:21 +0100 :ERROR : Cannot connect to node2 with SSH
2021-11-19 10:06:22 +0100 :FAILED : Verify SSH equivalence for the root user to node(s)
2021-11-19 10:06:22 +0100 :ERROR : FAILED run of command:/stage/dbserver_patch_21.211018/patchmgr -dbnodes /stage/nodes.conf -precheck -iso_repo /stage/20.1.15.0.0.211013/exadata_ol7_20.1.15.0.0.211013_Linux-x86-64.zip -target_version 20.1.15.0.0.211013 -allow_active_network_mounts -nomodify_at_prereq --log_dir /stage/logs
2021-11-19 10:06:22 +0100 :INFO : Precheck attempted on nodes in file /stage/nodes.conf: [node1 node2]
2021-11-19 10:06:22 +0100 :INFO : For details, check the following files in /stage/logs:
2021-11-19 10:06:22 +0100 :INFO : - <dbnode_name>_dbnodeupdate.log
2021-11-19 10:06:22 +0100 :INFO : - patchmgr.log
2021-11-19 10:06:22 +0100 :INFO : - patchmgr.trc
2021-11-19 10:06:22 +0100 :INFO : Exit status:1
2021-11-19 10:06:22 +0100 :INFO : Exiting.
Analysis
As first step, I checked the connection to the nodes. All nodes could be accessed without issues.
$> ssh root@node1 date
Fri Nov 19 11:03:05 CET 2021
$> ssh root@node2 date
Fri Nov 19 11:03:09 CET 2021
So far, every thing looked fine. The next step was to check the generated logs (I used -log_dir parameter to define my own location).
$> ls -l /stage/logs
total 8340
-rw-rw-r-- 1 oracle oinstall 4206395 Nov 24 13:22 Diag_patchmgr_dbnode_precheck_241121131024.tbz
-rw-rw-r-- 1 oracle oinstall 4298 Nov 24 13:22 ExadataImageNotification.pl.log
drwxrwsr-x 2 oracle oinstall 4096 Nov 24 13:10 notifications
-rw-rw-r-- 1 oracle oinstall 124905 Nov 24 13:22 patchmgr.log
-rw-rw-r-- 1 oracle oinstall 1690 Nov 24 13:22 patchmgr.stderr
-rw-rw-r-- 1 oracle oinstall 9037 Nov 24 13:22 patchmgr.stdout
-rw-rw-r-- 1 oracle oinstall 520953 Nov 24 13:22 patchmgr.trc
drwxrwsr-x 2 oracle oinstall 4096 Nov 24 13:22 tmp
-rw-rw-r-- 1 oracle oinstall 1 Nov 24 13:22 _wip_stderr
In the file patchmgr.stderr, I found the following strange syntax errors.
[40;1;36m================PatchMgr run started 2021-11-19 10:09:35 +0100 ===========[0m
File "/stage/dbserver_patch_21.211018/dcli", line 196
except IOError, (errno, strerror):
^
SyntaxError: invalid syntax
File "/stage/dbserver_patch_21.211018/dcli", line 196
except IOError, (errno, strerror):
^
SyntaxError: invalid syntax
[40;1;36m================PatchMgr run ended 2021-11-19 10:09:38 +0100 ===========[0m
Okay, there are some syntax errors. I opened a SR to analyze this.
Solution
With the help of Oracle Support, the root cause could be identified. The problem was caused by a non-compatible Python version. A colleague added a location with Python 3 to our PATH variable. After switching back to Python 2, everything was working again.
$> python -V
Python 3.6.8
$> export PATH=/bin:PATH
$> python -V
Python 2.7.5