Thursday 5 February 2015

Maintaining Tempfile in TEMP Tablespace of PDB$SEED in Oracle 12c (12.1.0.2)

During testing recovery procedures for one of the ongoing projects I wanted to test the "complete disaster" recovery scenario. In this scenario I had to recreate also all ASM disks and restore everything from backup.
Actually full backup with RMAN and subsequent restore of a pluggable 12c single-tenant database  was the solution. I will not talk about that as the main point of this post is quite different.

So the recovery was successful but after restoring  the CDB$ROOT and PDB database I found in the alert log the following message:


Errors in file /u01/app/oracle/diag/rdbms/mydb/mydb/trace/mydb_dbw0_28973.trc:
ORA-01157: cannot identify/lock data file 202 - see DBWR trace file
ORA-01110: data file 202: '+DATA/MYDB/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012015-01-03_12-22-30-pm.dbf'
ORA-17503: ksfdopn:2 Failed to open file +DATA/MYDB/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012015-01-03_12-22-30-pm.dbf
ORA-15173: entry 'FD9AC20F64D244D7E043B6A9E80A2F2F' does not exist in directory 'MYDB'
Thu Feb 05 09:12:52 2015
Errors in file /u01/app/oracle/diag/rdbms/mydb/mydb/trace/mydb_dbw0_28973.trc:
ORA-01186: file 202 failed verification tests
ORA-01157: cannot identify/lock data file 202 - see DBWR trace file
ORA-01110: data file 202: '+DATA/MYDB/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012015-01-03_12-22-30-pm.dbf'



The problem was the temp file of the temporary tablespace TEMP in PDB$SEED which is used for creation of pluggable databases. The tempfiles are not part of RMAN backup as they can always be recreated. Although I could live with this error messages as this is single-tennat database with only one pluggable database I wanted to deliver a "clean solution" to the client. Besides I wanted to know how I can solve such problem.

So here is the solution:

$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Thu Feb 5 09:35:11 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options

SQL> alter session set container=pdb$seed;

Session altered.

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY


SQL> select name from v$tempfile;

NAME
---------------------------------------------------------------------------------------------------
+DATA/MYDB/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012015-01-03_12-22-30-pm.dbf



SQL> alter pluggable database pdb$seed close;
alter pluggable database pdb$seed close
*
ERROR at line 1:
ORA-65017: seed pluggable database may not be dropped or altered


SQL> alter session set "_oracle_script"=TRUE;

Session altered.

SQL> alter pluggable database pdb$seed close;

Pluggable database altered.

SQL> alter pluggable database pdb$seed open read write;

Pluggable database altered.

SQL> alter tablespace temp add tempfile '+DATA';

Tablespace altered.
SQL> column name format a100
SQL> select name from v$tempfile;

NAME
-------------------------------------------------------------------------------------------------
+DATA/MYDB/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012015-01-03_12-22-30-pm.dbf
+DATA/MYDB/0BBEAC43B4250445E0530A0C6D0AEC65/TEMPFILE/temp.299.870860375

SQL>  alter tablespace temp drop tempfile '+DATA/MYDB/FD9AC20F64D244D7E043B6A9E80A2F2F/DATAFILE/pdbseed_temp012015-01-03_12-22-30-pm.dbf';


Tablespace altered.

SQL>  select name from v$tempfile;

NAME
----------------------------------------------------------------------------
+DATA/MYDB/0BBEAC43B4250445E0530A0C6D0AEC65/TEMPFILE/temp.299.870860375



After performed change there were no more error messages during database startup.

The secrete sauce for changing the state of PDB$SEED was setting "alter session set "_oracle_script"=TRUE;". The idea came from this post about changing the open mode of the PDB$SEED database.
Hope this short post will help somebody to resolve his problem with dropping and creating new tempfile in PDB$SEED faster.

Wednesday 28 January 2015

Finding Oracle VM Manager Core Management Port After OVM Manager Installation

Recently I was working on integration of OVM Manager with Oracle EM Cloud Control 12cR4 and I had to enter the "Oracle VM Manager URL" a.k.a. as "Core management port" which is reported at the end of OVM Manager installation. The default value for this port is 54321. As I was not in position to get this post-installation report because the installation was performed by another company, I had to find out the right port. The definition should be entered in the form: tcp://:.

EM Cloud Control and OVM Manager Integration Screen


OVM Manager Installation report sample


One of the the first ideas was to check local firewall configuration and see if there is anything for port 54321.

# iptables -L | grep 5432
ACCEPT     tcp  --  anywhere  anywhere   state NEW tcp dpt:54322

Search showed opened port 54322 which could also be one of the possibilities.
Searching with netstat revealed no acctivity on port 54322

# netstat -vatn | grep 54322

However, a lot of rows was returned when searching for port 54321:

# netstat -vatn | grep 54321
tcp        0      0 ::ffff:127.0.0.1:54321      :::*                        LISTEN
tcp        0      0 ::ffff:127.0.0.1:36797      ::ffff:127.0.0.1:54321      ESTABLISHED
tcp        0      0 ::ffff:127.0.0.1:51828      ::ffff:127.0.0.1:54321      ESTABLISHED
tcp        0      0 ::ffff:127.0.0.1:53096      ::ffff:127.0.0.1:54321      ESTABLISHED
tcp        0      0 ::ffff:127.0.0.1:55461      ::ffff:127.0.0.1:54321      ESTABLISHED
tcp        0      0 ::ffff:127.0.0.1:53017      ::ffff:127.0.0.1:54321      ESTABLISHED



But I was still not 100% sure, if the iptables showed the right port. So further digging through the documentation revealed the right port. The definition of the "Core management port"  is hidden in .config file under the name COREPORT in OVM Manager home directory /u01/app/oracle/ovm-manager-3/.config

The contents of this file in my case was:

# cat  /u01/app/oracle/ovm-manager-3/.config

DBTYPE=MySQL
DBHOST=localhost
SID=ovs
LSNR=49500
OVSSCHEMA=ovs
APEX=8080
WLSADMIN=weblogic
OVSADMIN=admin
COREPORT=54321
UUID=0004fb000001000075e54bae74172d82
BUILDID=3.3.1.1065


The explanation for the configuration items can be found in chapter "5.1 Backing up and Restoring Oracle VM Manager"  and obviously in my case the port used was actually the default one 54321.
The .config file should be part of the backup.

Hope this post will shorten search time for this kind of information to my readers.

Tuesday 27 January 2015

Using Hyphen In OS Username Or ORACLE_HOME Path in Oracle 12c Database Installation Will Make You Cry

Recently I was involved in an upgrade project on Linux platforms to Oracle 12c (12.1.0.2) and the customer wanted to name the owners of the Oracle SW distribution according to their company standard. So instead of using "grid"  and "oracle" OS account names they wanted to have the names which would start with "svc-" and continue with unique non Oracle standard name for either grid infrastructure or database software. Besides using this names as owners of grid and database software we used them also in the names of the GRID_HOME and ORACLE_HOME directories according to Oracle OFA suggestion.

So no problems with those names during installation and database creation. However, when  we wanted to patch the grid infrastructure and Oracle home with latest patches, the Opatch utility returned with an error that it can't parse the path to Oracle grid infrastructure. I immediately recognized the problem and the only remedy was to uninstall both grid and database SW. And then bang, another surprise, the Oracle 12c deinstallation procedures crashed as well. This time the reason was in checking the ownership of the distribution (grid or oracle).

The regular expression used to find the owner of the distribution in the script is /.*?\((\w+)\).*/
String returned by "id" command is:

uid=211(svc-xxxyyy) gid=201(oinstall) groups=201(oinstall), 200(dba), 205(asmadmin), 206(asmdba), 207(asmoper), 110000513(domain users) 

The $1 returned variable gets value "oinstall" instead of svc-xxxyyy and this causes deinstalltion script to fail.  The right expression would be /.*?\((\w+\-\w+)\).*/


So how one can manually deinstall database home and grid infrastructure home? For ORACLE_HOME deinstallation one can follow Doc.ID 1624580.1 and run:

cd $ORACLE_HOME/oui/bin
./runInstaller -silent -detachHome ORACLE_HOME="/u01/app/svc-xxxyyy/product/12.1.0/dbhome_1" ORACLE_HOME_NAME="OraDB12Home1"


This operation succeeded without any problems. Deinstalling GRID_HOME was slightly more complicated and actually at the moment of writing this post there is no document on MOS relevant for manual deinstallation of grid infrastructure with configured Oracle Restart. The suggestion from Oracle support was to follow "How to Deinstall Oracle Clusterware Home Manually ( Doc ID 1364419.1 )" which was relevant for 11gR2 but not 12c. Actually I have performed the deinstalltion in two steps:

# roothas.sh -deconfig -force
$ runInstaller -detachHome -silent -local ORACLE_HOME=$ORACLE_HOME


The $ORACLE_HOME in this case was actually pointing to GRID_HOME.

Why I said in the title that this will make you cry - because you have to start it all over again and the Oracle Installer doesn't warn you about the problematic hyphen  during the installation process when all prerequisites are run.

So after being working so many years with Oracle products I was really angry with myself that I was caught on such stupid mistake.