ALERT: opatch util cleanup deletes OS files in 12.2.0.1.19

With the release of the January patches, Oracle released version 12.2.0.1.19 of OPatch. In this specific release, the execution of opatch util cleanup deletes OS files outside the Oracle home directory structure. Thanks to my colleague Miguel for pointing this out.

Table of Contents

1 Problem
2 Update I
3 Workaround
4 Solution

Problem

Under normal circumstances, opatch util cleanup is used to clean up the $ORACLE_HOME/.patch_storage directory. The following output is from the version 12.2.0.1.17.

$> $ORACLE_HOME/OPatch/opatch util cleanup
Oracle Interim Patch Installer version 12.2.0.1.17
Copyright (c) 2020, Oracle Corporation.  All rights reserved.


Oracle Home       : /u00/app/oracle/product/19
Central Inventory : /u00/app/oraInventory
   from           : /u00/app/oracle/product/19/oraInst.loc
OPatch version    : 12.2.0.1.17
OUI version       : 12.2.0.7.0
Log file location : /u00/app/oracle/product/19/cfgtoollogs/opatch/opatch2020-01-28_11-24-01AM_1.log

Invoking utility "cleanup"
OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
You will be still able to rollback patches after this cleanup.
Do you want to proceed? [y|n]
y
User Responded with: Y

Backup area for restore has been cleaned up. For a complete list of files/directories
deleted, Please refer log file.

OPatch succeeded.

$> cat /u00/app/oracle/product/19/cfgtoollogs/opatch/opatch2020-01-28_11-24-01AM_1.log
[Jan 28, 2020 11:24:02 AM] [INFO]   CUP_LOG: Trying to load HomeOperations object
[Jan 28, 2020 11:24:02 AM] [INFO]   CUP_LOG: HomeOperations object created. CUP1.0 is enabled
[Jan 28, 2020 11:24:02 AM] [INFO]   OPatch invoked as follows: 'util cleanup -invPtrLoc /u00/app/oracle/product/19/oraInst.loc '
[Jan 28, 2020 11:24:02 AM] [INFO]   Runtime args: [-Xmx3072m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/u00/app/oracle/product/19/cfgtoollogs/opatch, -DOPatch.ORACLE_HOME=/u00/app/oracle/product/19, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=/u00/app/oracle/product/19/OPatch, -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/u00/app/oracle/product/19/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
[Jan 28, 2020 11:24:02 AM] [INFO]   Heap in use : 21 MB
                                    Total memory: 117 MB
                                    Free memory : 95 MB
                                    Max memory  : 2731 MB
[Jan 28, 2020 11:24:02 AM] [INFO]   Oracle Home       : /u00/app/oracle/product/19
                                    Central Inventory : /u00/app/oraInventory
                                       from           : /u00/app/oracle/product/19/oraInst.loc
                                    OPatch version    : 12.2.0.1.17
                                    OUI version       : 12.2.0.7.0
                                    OUI location      : /u00/app/oracle/product/19/oui
                                    Log file location : /u00/app/oracle/product/19/cfgtoollogs/opatch/opatch2020-01-28_11-24-01AM_1.log
[Jan 28, 2020 11:24:02 AM] [INFO]   Patch history file: /u00/app/oracle/product/19/cfgtoollogs/opatch/opatch_history.txt
[Jan 28, 2020 11:24:04 AM] [INFO]   [OPSR-TIME] Loading raw inventory
[Jan 28, 2020 11:24:05 AM] [INFO]   [OPSR-MEMORY] Loaded all components from inventory. Heap memory in use: 21 (MB)
[Jan 28, 2020 11:24:05 AM] [INFO]   [OPSR-MEMORY] Loaded all one offs from inventory. Heap memory in use: 23 (MB)
[Jan 28, 2020 11:24:05 AM] [INFO]   [OPSR-TIME] Raw inventory loaded successfully
[Jan 28, 2020 11:24:05 AM] [INFO]   Invoking utility "cleanup"
[Jan 28, 2020 11:24:05 AM] [INFO]   [OPSR-TIME] Cleaning up backup
[Jan 28, 2020 11:24:05 AM] [INFO]   OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
                                    You will be still able to rollback patches after this cleanup.
                                    Do you want to proceed? [y|n]
[Jan 28, 2020 11:24:05 AM] [INFO]   Start to wait for user-input at Tue Jan 28 11:24:05 CET 2020
[Jan 28, 2020 11:24:08 AM] [INFO]   Finish waiting for user-input at Tue Jan 28 11:24:08 CET 2020
[Jan 28, 2020 11:24:08 AM] [INFO]   User Responded with: Y
[Jan 28, 2020 11:24:08 AM] [INFO]   [OPSR-TIME] Loading cooked inventory
[Jan 28, 2020 11:24:08 AM] [INFO]   [OPSR-MEMORY] : Loading cooked one offs. Heap memory used 24 (MB)
[Jan 28, 2020 11:24:09 AM] [INFO]   [OPSR-MEMORY] : Loaded cooked oneoffs. Heap memory used : 113 (MB)
[Jan 28, 2020 11:24:09 AM] [INFO]   [OPSR-TIME] Cooked inventory loaded successfully
[Jan 28, 2020 11:24:09 AM] [INFO]   CUP_LOG: Found poh CUP 29834717 is a subset of other poh CUP: 30125133
[Jan 28, 2020 11:24:10 AM] [INFO]   CUP_LOG: Found poh CUP 29834717 is a subset of other poh CUP: 30125133
[Jan 28, 2020 11:24:11 AM] [INFO]   [OPSR-TIME] Backup area for restore has been cleaned up. For a complete list of files/directories
                                    deleted, Please refer log file.
[Jan 28, 2020 11:24:11 AM] [INFO]   Finishing UtilSession at Tue Jan 28 11:24:11 CET 2020

But with 12.2.0.1.19 opatch deletes files (or at least it tries) outside the Oracle home. Which directories are cleaned up depends on the current working directory. For example, when the current directory is set to $ORACLE_BASE, opatch deletes files from /etc.

Excerpt from the logfile:

...
[Jan 28, 2020 11:29:43 AM] [INFO]   Invoking utility "cleanup"
[Jan 28, 2020 11:29:43 AM] [INFO]   [OPSR-TIME] Cleaning up backup
[Jan 28, 2020 11:29:43 AM] [INFO]   OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
                                    You will be still able to rollback patches after this cleanup.
                                    Do you want to proceed? [y|n]
[Jan 28, 2020 11:29:43 AM] [INFO]   Start to wait for user-input at Tue Jan 28 11:29:43 CET 2020
[Jan 28, 2020 11:29:47 AM] [INFO]   Finish waiting for user-input at Tue Jan 28 11:29:47 CET 2020
[Jan 28, 2020 11:29:47 AM] [INFO]   User Responded with: Y
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/kshrc" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/crypttab" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/mtab" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/fonts/conf.d/65-0-madan.conf" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/fonts/conf.d/59-liberation-sans.conf" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/fonts/conf.d/65-0-lohit-kannada.conf" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/fonts/conf.d/59-liberation-mono.conf" could not be deleted
[Jan 28, 2020 11:29:47 AM] [INFO]   File "/etc/fonts/conf.d/61-urw-bookman.conf" could not be deleted
...

Calling opatch from the $ORACLE_HOME is even worse. It deletes files from /etc, /bin and much more.

Excerpt from the logfile:

...
[Jan 28, 2020 11:39:27 AM] [INFO]   Directory "/lib/debug/lib64" could not be deleted (FAILURE)
[Jan 28, 2020 11:39:27 AM] [INFO]   Directory "/lib/debug/sbin" could not be deleted (FAILURE)
[Jan 28, 2020 11:39:27 AM] [INFO]   Directory "/lib/debug" could not be deleted (FAILURE)
[Jan 28, 2020 11:39:27 AM] [INFO]   File "/lib/systemd/system-preset/85-display-manager.preset" could not be deleted
[Jan 28, 2020 11:39:27 AM] [INFO]   File "/lib/systemd/system-preset/90-default.preset" could not be deleted
...

Even when the most part of files is not deletable by the oracle user, it can have an impact on backup tools or the startup of the Grid Infrastructure.

After a patch is applied using opatch/opatchauto apply, the cleanup procedure is executed. At least it looks like that this implicit call of util cleanup works as expected.

Update I

I did some more tests and the behavior is really strange. When you execute opatch from the root directory it fails with a java exception.

$> cd /
$> $ORACLE_HOME/OPatch/opatch util cleanup
...
UtilSession failed: java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/dev/fd/49/proc/self/map_files: Operation not permitted

OPatch failed with error code 73

But after that, every subsequent execution from Oracle Base or Oracle Home succeeds without trying to delete files from /etc, /bin, etc.

Workaround

At the moment I am not aware of a workaround. So the only solution is to use the previous version 12.2.0.1.17 of opatch.

Solution

Bug 30362460 – OPATCH UTIL CLEANUP TRIES TO RECURSIVELY DELETE ALL FILES IN ROOT DIRECTORY INCLUDING /LIB was raised which addresses the described error. It will be delivered with the next OPatch release. Until the release of this patch, don’t use opatch util cleanup.