19.3 PDB Close ORA-65107 ORA-16078

When closing a cloned PDB in 19.3, I am receiving the following errors:

ORA-65107: Error encountered when processing the current task on instance:1
ORA-16078: media recovery disabled

This may exist in 12.2 or 18c, but I skipped those versions. If anyone has info if this is an issue in the versions I skipped, comment below.

I tracked this down the fact that I’m closing the cloned PDB with ABORT. If I close IMMEDIATE, I do not get the error. Here is an example of where I received the error twice with ABORT but not with IMMEDIATE, so I do have my workaround.

SQL> alter pluggable database dba1 close abort instances=all;
alter pluggable database dba1 close abort instances=all
*
ERROR at line 1:
ORA-65107: Error encountered when processing the current task on instance:1
ORA-16078: media recovery disabled

SQL> alter pluggable database dba1 close immediate instances=all;

Are these physical therapies safe? generic professional viagra Physical therapies are very safe if they are applied by expert hands and at the right times. These side-effects are very common and stay buy cialis go to this pharmacy for very short time but in rare cases, it might develop suddenly. You can cialis sildenafil purchase any of the products that are less effective and fake here; all are approved from the USFDA. However, sometimes it so happens that cialis viagra the organ is too stressed or strained, or even too weak due to medical reasons.

Pluggable database altered.

SQL> alter pluggable database dba1 open instances=all;

Pluggable database altered.

SQL> alter pluggable database dba1 close abort instances=all;
alter pluggable database dba1 close abort instances=all
*
ERROR at line 1:
ORA-65107: Error encountered when processing the current task on instance:1
ORA-16078: media recovery disabled

SQL> alter pluggable database dba1 close immediate instances=all;

Pluggable database altered.

ORA-01173 Opening PDB in Upgrade Mode

I am working on upgrading a Multitenant environment from Oracle 12.1.0.2 to Oracle 19c. I have 40 PDBs supporting all sorts of dev and test environments. We typically upgrade a few dev databases first and then follow with test databases before upgrading production. When production is upgraded, we like to keep a few dev databases around at the old version for a period of time. This way, if someone thinks the database upgrade caused a problem, we can repeat the test in the old version to verify. All of this is a long way of saying that I cannot upgrade the CDB along with all of its PDBs at the same time.

To upgrade, I chose to prepare the PDB for upgrade and unplug it form the 12.1.0.2 CDB. The PDB is then plugged into a 19.3 CDB, opened in upgrade mode and then upgrade. All of this is documented here.

So the PDB is prepacked and unplugged form the old version CDB. The PDB is then plugged into the new version CDB and opened in Upgrade mode when I get this error:

SQL> alter session set container=my_PDB;

Session altered. 

SQL> alter pluggable database open upgrade; 
alter pluggable database open upgrade 

ERROR at line 1: 
ORA-01173: data dictionary indicates missing data file from system tablespace 

The side effects associated with discount pfizer viagra browse around that include diarrhea, stomach upset, and headache, running nose and flushing and dizziness. An ED treatment not only requires a medical solution, but it also needs many vital things to ensure the buying that cialis prescription safe treatment. But soon after the launch of series of treatments for impotency this miss-conception has been denied to exist anymore in man’s viagra online canada life. viagra online consultation Our body utilizes bile acids from the bile to make fat droplets smaller.

Oracle Support told me the ORA-1173 error indicates corruption in the Data Dictionary. This never made any sense to me. The PDB does not contain any data dictionary for the data files. That is stored in the CDB. If you query DBA_DATA_FILES in the PDB, its just a pass-through to the CDB filtering out other files not belonging to that PDB. When the PDB is unplugged, the datafile information is written to a XML file. If there is any dictionary corruption it would occur when the XML file is generated or when the PDB is plugged in. Yet we verified the XML file’s contents are accurate and so is the Data Dictionary contents after being plugged in. The ORA-1173 is not accurate to the root cause of my problem.

After three months of working with Oracle Support, I never obtained a resolution. However, I have devised a workaround. I received the ORA-1173 error when the 19c CDB was created fresh at that version. However, when I created a new, empty 12.1.0.2 CDB and upgraded it to 19.3, I did not receive the ORA-1173 error when opening a plugged-in PDB in upgrade mode. So that is my workaround. Create the new CDB then upgrade it to the target version.

Oracle Documentation

Oracle has changed their documentation, making it much harder to find the Oracle Database docs. To access the docs, go to http://docs.oracle.com. From there, you could click on the Database category. As it stands now, the only icons on the doc main page are all cloud-related.

It took me a few minutes to figure out where they buried the Database documentation. Click on the three lines in the top left corner to pull down the menu. Then select Database –> Oracle Databases.

For the high buy cialis price of the medicine starts in an hour and lasts till 4 to 6 hours. Another option is viagra no prescription uk to go through a live class at a commercial driving school or through close friends she knows and trusts. The erection can last for 4 to pamelaannschoolofdance.com cialis generika 6 hours. Moreover, user must also discuss with health care provider if sipping other prescribe or over the counter drugs. purchase viagra online pamelaannschoolofdance.com being one of these All in all, the strain in life and overly busy life won’t enable peaceful recovery.

asdf

GI 19c RPM Package Manager Database

I am upgrading an Oracle 18c Grid Infrastructure cluster to the new Oracle 19c version, release for on-premises last week. The OUI pre-requisite checks finds two issues that need my attention. The first issue is that I am missing Patch 28553832 which should be easy to solve. Simply download the patch and apply it before attempting this upgrade. The second issue says “RPM Package Manager database”. What is this? To learn more, I clicked on the Details link for this finding. You can see the information in the screenshot below.

As we can see, I do not have any credentials for the ‘root’ user so the OUI is having a problem verifying the RPM Package Manager on my system. The solution is easy enough. Press the Back button in the OUI to arrive at the screen where I can have the OUI automatically run the root scripts for me.

There are even a few young men about 20 who experience from issues, but normally you will buying viagra from india not listen to about it. For some, this infertility clinic consequence viagra tablets price might be a good idea for a man to cut the pill in half so you can take it separately. Even Bill Gates, the founder of giant Microsoft Corp., had no escape from the BSOD error during the beta release of Windows 98 at Computer levitra tablets Dealer’s Exhibition (COMDEX) in Las Vegas in April, 1998. Curing osteoporosis, preventing purchase cialis from india hardening of arteries and boosting oxygen consumption of cells are other health benefits of consuming horny goat weed.

Normally, I run the rootupgrade.sh script manually and I leave these fields blank. This time, I checked the box to run the config scripts automatically. I do not know the root password on this system but I do have sudo access so I enter in the details for the second option. I then press Next and have the OUI check the pre-requisites once again. This time, the check for the RPM Package Manager succeeds.

Personally, I like to have more manual control over my upgrade process and I like to run the rootupgrade.sh script myself. In this case, once I know the pre-req check passes, I can go back here and uncheck the box to auto run the root script. The pre-req check will fail again, but I can ignore it this time.

What do you do if you do not have the root password or sudo access? You can have your SysAdmin come to your workstation and type in the root password to verify the pre-req check passes. Then re-run the OUI and ignore the finding the next time. You will probably have to get your SysAdmin to run the rootupgrade.sh script when it is time to do so.

Oracle DBA Mentor

My new book is now available from Apress. Oracle DBA Mentor is different from most other Oracle books. The others will teach you some great things and show you how Oracle works and how you should use it. This book teaches you to teach yourself. Where do you turn when those books do not contain the answers you seek? This book is your mentor. Pick up a copy today!
https://www.apress.com/us/book/9781484243206

As it is completely natural, no prescription is necessary for the reason that you need to confirm you will view description free cialis samples keep away from junk foods. There are thousands of people who fail to please their women in the bedroom. order viagra viagra Finally, guys who just aren’t take a break from getting themselves off, even when your penis is chronically sore, irritated and even bruised – may need some Learn More Here levitra 40 mg help disengaging from harmful behavior. A health professional diagnosesthe cause and recommend the medication to improve the blood circulation in the veins and arteries that improve the stamina and strength becomes at the topmost level. levitra cheap online

ORA-65139: Mismatch between XML metadata file and data file

I was trying to plug a non-CDB into our new Multitenant environment as we make the move to Multitenant. I am going to create a golden image of our production non-CDB and then all dev and test databases will just be clones off the golden image. But first, I need to get the non-CDB plugged in. I have the disk snapshot mounted to the Multitenant database servers. I also generated the XML file and I’m ready to plug in the non-CDB with this command:

CREATE PLUGGABLE DATABASE gold180904
USING '/home/oracle/source_db.xml'
NOCOPY
SOURCE_FILE_NAME_CONVERT=('/u01/app/oracle/oradata/data01',
         '/u01/app/oracle/oradata/mt_golden_2018_09_06_095555/data01',
'/u01/app/oracle/oradata/data02','/u01/app/oracle/oradata/mt_golden_2018_09_06_095555/data02',
'/u01/app/oracle/oradata/data03','/u01/app/oracle/oradata/mt_golden_2018_09_06_095555/data03',
'/u01/app/oracle/oradata/data04','/u01/app/oracle/oradata/mt_golden_2018_09_06_095555/data04')
TEMPFILE REUSE;

Unfortunately, I ran into the following error:

CREATE PLUGGABLE DATABASE gold180904
*
ERROR at line 1:
ORA-65139: Mismatch between XML metadata file and data file
/u01/app/oracle/oradata/mt_golden_2018_09_06_095555/data03/datafile12.dbf for
value of rdba (4194824 in the plug XML file, 4458552 in the data file)

In my research, the ORA-65139 error is normally seen when the XML file was generated with the database open as READ WRITE. But I know for a fact that my database was READ ONLY when the XML file was generated.  Furhtermore, all of the similar issues I found on MOS and in Google searches all had “value of fcpsb” whereas the last line of my error message says “value of rdba”. Well I’m not sure what the RDBA has to do with this and neither of the values in the error message map to the datafile listing in the message. So this was a puzzler for me.
However, buying Kamagra online is a better alternative for the treatment viagra soft 50mg of impotence, especially if you’re after a long-term impact. cialis tablets 100mg No matter a man suffers from the type, it is always humiliating for him not satisfying his female partner. You have the freedom to choose, so take advantage of it. pdxcommercial.com viagra price Please tell your doctor regarding the conditions like sample viagra prescription diabetes, heart disorders, cancer or other associated conditions.
After trying a few different things, I decided to run the check for plugin compatiblity.

DECLARE
compatible BOOLEAN;
BEGIN
compatible:=DBMS_PDB.CHECK_PLUG_COMPATIBILITY(
     pdb_descr_file=>'/home/oracle/source_db.xml',
     pdb_name=>'GOLD180904');
END;
/

When querying PDB_PLUG_IN_VIOLATIONS, one line had an error.  Its message in that view said:

PSU bundle patch 180717 (DATABASE PATCH SET UPDATE 12.1.0.2.180717): Installed in the PDB but not in the CDB.

This now makes more sense. The source database is a production environment with the latest PSU applied. The CDB is brand new and has yet to see any patches. I applied the latest PSU to the CDB and the plugin operation worked successfully on the next try.

In the end, it was obvious the error message had nothing to do with the root cause of the problem. At least not to me.

RAC Sequence Contention

I recently ran into a case where selecting the next value from a sequence was causing contention issues in Oracle RAC.  See this screen shot from Lighty (click on the image to see a larger picture)

The wait events will look the same if viewed in Enterprise Manager’s performance screens, which does require one to license the optional Diagnostics Pack.
Some patients viagra sale in canada should also not take it because of their medicines. prices of viagra Men who suffer from diabetes may also suffer from poor quality erections, or just not be able to look this young and beautiful if they didn’t were extremely conscious about their lifestyle. purchasing viagra The caffeine present in dark chocolate enhances energy, supply bone strength, reduces fatigue and offers performance-efficacy. This djpaulkom.tv purchase generic levitra claim is not yet clear until now.
We can see high waits on the row cache lock wait event as well as multiple global cache wait events (all start with “gc”).

The problem was the sequence was created with CACHE set to zero. Sequences in Oracle RAC with a cache setting too low will see wait events like this. The solution is simple, increase the CACHE size.

PRVG-2027 Owner of file is inconsistent across nodes

I recently ran into an issue right after I applied the latest/greatest CPU to an Oracle RAC system. Both GI and RAC were patched. The GI alert log showed error messages like:

2018-05-13 22:03:01.121 [SRVM(32264)]CRS-10051: CVU found following errors with Clusterware setup : PRVG-2027 : Owner of file “/pkg/grid/crs12.1.0.2/lib/ libclntsh.so.12.1” is inconsistent across nodes. [Found = “{root=[host50], oracle=[host51]}” on Nodes = “host51,host50”]

The file above was not the only one with this error. After analysis, host50 is correct and host51 has the incorrect file ownership.

Fixing this required downtime and I needed to run the following commands:
The nitric oxide causes an enzyme, guanylate cyclase, viagra samples for free to produce cyclic guanosine monophosphate (cGMP). purchase levitra online learningworksca.org The delivery service then takes over and makes sure that you face a proper blood supply to the penis. The known ranges have historically read as: Normal: Less than 120 over 80 Prehypertension: 120 to 139 over 80 to 89 Stage 1 high BP: 140 to 159 over 90 to 99 Stage 2 high BP: 160 and above over 100 and above High blood pressure in people over 60 years: 150 and above over 90 and above Categories of High BP Essential condition which is common in Multiple Sclerosis can lead to male. have a peek here cheap viagra So, they should take sildenafil österreich proper initiative to cure weakness due to excessive nightfall.
$GRID_HOME/crs/install/rootcrs.sh -unlock

$GRID_HOME/crs/install/rootcrs.sh -patch

Running those commands will stop then start Grid Infrastructure and fix the file ownership and any permissions.

Large .patch_storage

I received an alert from Enterprise Manager that one of my production databases was getting low on disk space. I tracked it down to $GRID_HOME/.patch_storage which was consuming 30GB of my 90GB drive. Yikes!

The first thing I did was to run the opatch cleanup routine as I documented here back in 2013: http://www.peasland.net/2013/03/21/patch_storage/

Unfortunately, it didn’t clean up anything.

This time, I had to resort to a manual cleanup. Here are the steps I did.

The files in .patch_storage start with the patch molecule number and a timestamp. For example: 19582630_Nov_14_2014_21_43_23

I need to ask opatch if that patch is still in the inventory.

$ORACLE_HOME/OPatch/opatch lsinventory|grep 19582630
20075154, 20641027, 22271856, 20548410, 19016964, 19582630

lsinventory shows the patch is in the inventory. I move on to the next patch.
The kamagra buy generic cialis jelly has been brought in many delicious flavors such as orange, strawberry, mint, chocolate, banana etc. Apart from this High sildenafil tablets 100mg blood pressure affects Hormone balance and nitric acid levels. Causes for Erectile Dysfunction: Numbers of causes play a significant role for arising the problem of oligospermia in males, but had we ever discussed about the premature orgasm in women and shared everything in detail. cialis viagra sale Benefits of Ayurveda Brimhana Therapy or healthy weight gain therapy: The person who online viagra prescription undergoes weight gain therapy enjoys the following benefits- The body mass increases.
When my lsinventory command returns nothing, the patch is not in the inventory.  MOS Note 550522.1 says you can remove that directory as its  no longer needed.  The ever-cautious DBA personality in me wants to ensure I can recover from a simple “rm -rf dir_name” command. So I tar and gzip the directory first, then remove the directory.

tar cvf 25869825_Jul_3_2017_23_11_58.tar 25869825_Jul_3_2017_23_11_58

gzip 25869825_Jul_3_2017_23_11_58.tar

rm -rf 25869825_Jul_3_2017_23_11_58

Its painstaking work doing this for each and every patch. I’m sure someone who is better than me with sed and awk and shell scripting could automate this process.

By following these steps, my .patch_storage directory dropped from 30GB down to 11GB.

Next quarter when I apply my CPU again, should opatch cry foul and demand these be placed back, I can quickly unzip and extract the tarball and opatch should be happy.

I did this operation on $GRID_HOME but it will work on $RDBMS_HOME as well. Also, since this is Oracle RAC, I may want to do this on all nodes in the cluster.

Where are my Patches?

I thought I had posted something like this on my blog previously but I couldn’t find it when I was looking for someone earlier today. So here is the information anew. If my old blog post is not more than just a figment of my imagination, it probably needs updating anyway. 🙂

How do I find the latest/greatest patches for my database? Well the answer is often to apply the most recent quarterly Cumulative Patch Update (CPU). All too often on the MOSC and OTN forums, I see people who ask where the most recent CPU is and someone provides the patch number for them. That’s great but in 3 months, the info will be out of date. Here is how you can find your most recent patches for currently supported versions.

First, point your browser to OTN (http://otn.oracle.com) then in the Essential Links section, click on Critical Patch Updates.

The next page shows you each quarter’s CPU patches. Near the top of the page is a section showing you the CPUs. One thing I like about this page is that it shows you the next four release dates. In my screen shot below, the next CPU will be released on 18 Jan 2018. This is great for planning. As I write this on 8 Nov 2017, I know that I’m still more than 2 months away. But if I were looking for the most recent CPU and today were 17 Jan 2018, I’d probably just wait one more day to download the next newest CPU and be even more current.

The table that follows contains a link to each CPU. You can always find the most recent one at the top of the table. In the screenshot above, the Oct 2017 CPU can be seen as the most recent. Almost every time, you will be downloading the most recent CPU. But on rare occasions, you may be interested in one from the past. Click on the link under the Critical Patch Update page of the quarter you need.
This is because people who are not even suffering from the erectile dysfunction can also take an interest http://www.glacialridgebyway.com/windows/Old%20Log%20Church.html buy viagra sale in assuming this particular role. “The Lost Child” deals with his problems by choosing to remain invisible. Aged people sildenafil tab tend to face this disorder as well. Ginger raises the body temperature and sensitivity of the male organ. viagra viagra Impotency has viagra online no prescriptions the power to create havoc in your life.
This will bring you to a page with a big table showing links to every possible CPU for every possible Oracle product. To get to the CPU, you will click on the link under the Patch Availability column. If you click on the link under the first column, as I have done more times than I’d like to admit, you will not end up in the correct place.

I typically only deal with the Database. So I scroll down the list of products until I find the Oracle Database Server line and then click on the Database link on the right.

That’s it, you have now arrived at the MOS Note for that quarter’s CPU for your product.

Happy patching!