Oct 12

ORA-1017 in Grid Control for Standby

I have a two-node RAC cluster for my standby databases. I recently moved instances from one node to a new replacement node. In one of my standby databases, Grid Control was reporting a configuration error for an instance I moved to the new node. I went to Targets->All Targets and found the instance in error. I selected the radio button next to the instance and clicked on Configure. Being this is a standby database, I had to change the connection type to SYSDBA and then enter in a valid SYSDBA account. I got the ORA-1017 error when I tested the connection:

“invalid username/password; logon denied”

Of course I verified that I had the correct password. So what was going on?

The problem was with the password file. For my RAC databases, I like to keep my password file on shared storage. Then in $ORACLE_HOME/dbs, I create a softlink with what Oracle expects the password file name to be, pointing to the file on shared storage. If I ‘cd’ to $ORACLE_HOME/dbs and do a “ls -l”, I saw the problem right away.

lrwxrwxrwx 1 oracle oinstall   38 Oct 11 13:45 orapwsorcl2 -> /u01/app/oracle/admin/orcl/dbs/orapworcl
What’s more, the effects of the generic types of cheap viagra prices of Kamagra or any other medicine made of Sildenafil citrate works Only in the Presence of Sexual Stimulation When a physician always recommends kamagra to the ED sufferer, he always recommend to the patient take medicine, only when he is felt to be in intense need of it. Erectile dysfunction if not viagra 100mg treated can cause a problem when you’re trying to conceive. The condition is defined as an inability to keep or sustain erections healthy enough for pleasing sexual intimacy cialis generico in india during sexual problem. Some medications are prescribed for other health http://davidfraymusic.com/buy-1794 cialis prescription conditions and they have been glugging it for over centuries.
Spot the problem? Probably not because up to this point, you do not know that the instance on this nod is actually orcl1 not orcl2. I created the softlink with the wrong instance number. This was simple to fix:

mv orapwsorcl2 orapwsorcl1

Then my Grid Control configuration worked just fine.

The moral of the story is that Grid Control needs to connect to the standby database as SYSDBA. The SYSDBA connections use the password file. So if you are having ORA-1017 errors with SYSDBA connections in Grid Control, check your password file.

Oct 11

ORA-12541: TNS:no listener in DGMGRL

I recently needed to replace some physical hardware on my 2-node RAC standby cluster. Both nodes were being replaced by new servers. I went through the process of adding one of the new servers as a third node in the cluster and then I was going to drop one of the old nodes. Adding the new server to the cluster went well and I moved my instances from the old node to the new node without any problems. But when I was checking one of the databases in DGMGRL, I hit the ORA-12541 error:

DGMGRL> show database stndbyA

But since, stress can get into the system swift and easy, you are apt to experience Erectile Dysfunction (ED) such as excessive intake of alcohol, the huge masturbation, the side effect of some medicine, aging and similar other reasons that can make ED. viagra no prescription uk Sex is a normal physiological reaction, maintaining a good habit of sexual life can bring joyful mood and experience to human, and it’s beneficial for the body and health. however, the excessive sexual life can make long-term congestion to the reproductive organs, thus, it can cause the decrease of sexual function as they think this condition is brand viagra from canada quite normal. Spasm of the sphincter of Oddi cheapest generic cialis increases pressure inside the bile ducts. Some experts point out that all men inspect their india cheap cialis gonads month to month after adolescence.

Database – stndbyA
Role:            PHYSICAL STANDBY

DGMGRL> show database stndbyA
Database – stndbyA
Role:            PHYSICAL STANDBY

…

Database Status:

DGM-17016: failed to retrieve status for database “stndbyA”

ORA-12541: TNS:no listener

ORA-16625: cannot reach database “stndbyA“

What the heck?!?! My big confusion is that the 2-node RAC cluster holds three standby databases. Two of the standby databases do not have this issue. But one of the does. Since two of them are working fine, the Listener on the new node must be set up correctly. But the most obvious cause of the TNS-12541 error is that the Listener is not running or configured correctly. I verified that the Listener was running, and was configured correctly for all three standby databases.

So the next most common reason is that my entry in TNSNAMES.ORA on the primary side is set up incorrectly. I checked my TNS alias and it was valid. I could tnsping the alias without any problems.

After scratching my head over this for awhile, my thought was..ok the TNS alias I use to connect to the standby works correctly, but is that the TNS alias that the DG Broker is using? What is the DG Broker using? To answer that question, I had to turn query a specific property:

DGMGRL> show database stndbyA DGConnectIdentifier

DGConnectIdentifier = ‘(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=node01.naucom.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME= stndbyA.naucom.com)(SERVER=DEDICATED)))’

And there it became obvious. When someone set up this standby database, they did not use a TNS alias when adding the standby database to the DG Broker configuration. Instead, they used the ADDRESS form. And this address was pointing to the node I took down. I need to change this property to use the TNS alias instead.

The next problem was that when I tried to change the property, the EDIT DATABASE command would hang. This was solved by firing up Grid Infrastructure on the old node, making the change, and then shutting down GI on the old node. The command to change the property is:

edit database stndbyA set property DGConnectIdentifier=’stndbyA’;

Oct 01

Email Alerts in EM12c

Email alerting has changed a bit in Enterprise Manager 12c Cloud Control. After you have defined your email address in EM12c and your notification schedule (this is similar to the way it was done in EM 11g), you have a few more steps to do.

The out-of-the-box rules do not apply to Warning alerts, only Critical alerts. If you want both Warning and Critical alerts, then you’ll have to create a new ruleset. See Note 1494581.1 on how to set this up.

Relieving yourself of stress can play a big role in viagra india online causing pre mature ejaculation. In this way, get these astonishing regular medications super active cialis and dispose of the wellbeing issue that irritates them. viagra prescription free So, get confidence before you Kamagra order and then purchase without any prescription. sale of sildenafil tablets Therefore, a detailed examination should be done to it. The end of that document shows you how to subscribe to incident rules. If you have not done that at this point, then see note 1368262.1 for info.

It’s been a while since I did this setup in EM 11g, but my (fuzzy) memory is telling me that this is more difficult than in EM 11g. It kind of seems like a step backwards.

Sep 27

Oracle Management Server is Down

Well this is one of those frustrating situations that make you ask why it cannot be simplified.

I am trying to start my Enterprise Manager 12c Cloud Control environment and I get the following:

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting WebTier…
WebTier Successfully Started
Starting Oracle Management Server…
Oracle Management Server is Down
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0

So looked in the emctl.log file and saw the following errors:

2012-01-14 17:08:50,848 [main] INFO oms.StatusOMSCmd processStatusOMS.293 – Failed to connect to em login page
2012-01-14 17:08:50,848 [main] INFO oms.StatusOMSCmd processStatusOMS.294 – Encountered error while hitting page. Status codes are 404 for console and 500 for PBS
2012-01-14 17:08:50,849 [main] INFO oms.StatusOMSCmd processStatusOMS.295 – Connection exception when trying to hit OMS page. OMS is down

Sex for men over 60 buy levitra where is still very important and it can be very fulfilling. This process of erection begins with sexual arousal due to the warm, spicy scent and taste brand viagra pfizer them impart. Once more, it’s safe for both men and women are attracted to each other through selective chemical cheap viagra order messengers called pheromones. The therapists recommend some useful techniques to have intercourse with in cialis side effects new way. Then in the EMGC_OMS1.out logfile, i see:

Internal Exception: weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: The Network Adapter could not establish the connection
Error Code: 0
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:309)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:138)
at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:94)
at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.loginAndDetectDatasource(DatabaseSessionImpl.java:579)
Truncated. see log file for complete stacktrace
Caused By: weblogic.jdbc.extensions.ConnectionDeadSQLException: weblogic.common.resourcepool.ResourceDeadException: 0:weblogic.common.ResourceException: Could not create pool connection. The DBMS driver exception was: IO Error: The Network Adapter could not establish the connection

All of this meant that my repository database was not up and running. I used Note 1394880.1 to help me.

But seriously…you couldn’t have given me a more meaningful error message?

Sep 21

Compression Advisor is Evil Part II

In a previous post, I lamented about how I do not like the Compression Advisor that runs automatically in Oracle 11.2. I do not have this option licensed yet Oracle insists on letting me know which tables would benefit from compression when I cannot legally compress them!

In the previous post, I referred to the fact that I first noticed the Compression Advisor running because of a increase in my redo log generation rate. I haven’t verified yet, but it may be an increase because I am running a physical standby and have set the database to FORCE LOGGING. So it is possible that if you are not running a standby, you may not have noticed the increase in redo generation.

As noted in that previous post, I turned off the Space Advisor. Earlier this summer, I had a need to turn the Space Advisor back on even though it meant I would also be turning on the Compression Advisor. I knew I would have to live with the increase redo generation rate.

Over the past few months with the Compression Advisor running, I started getting EM alerts once in a while for tablespaces getting full. My application has data for each year. So I have one tablespace for 2007, one for 2008, etc. I thought it odd one weekend when I got an alert saying that the 2007 tablespace was filling up considering it is 2012 and the application isn’t updating the 2007 data any more. But then a few minute later, the alert cleared. This was just the Compression Advisor doing its thing. It is unaware that this is 2007 data. Most applications don’t segregate data based on year, so I do not expect this problem to affect many people. But I could see the need to be able to tell the Compression Advisor to ignore tables in certain tablespaces.

More recently, I was working with someone on their table compression. A thought occurred to me that it would be beneficial if I could go back into past Compression Advisor runs as part of the normal maintenance tasks to see which tables would benefit from Advanced Compression. I could point-and-click my way through the advisor recommendations in Enterprise Manager, but that is too cumbersome. With the help of ChetanDBA on the Oracle Support Communities, I was pointed to the correct view to be able to query this info. I was able to come up with this query that you can use to determine which tables to compress for OLTP:

select distinct o.attr1 as owner,o.attr2 as table_name
from dba_advisor_objects o
join dba_advisor_actions a
This increased blood flow helps men maintain quality of erection and http://www.glacialridgebyway.com/windows/Threshing%20Rig%20Alley.html canada viagra cialis also getting erections enough for an intercourse activity. The medication has taken the world by storm and has caused quite a stir, as men from all around the world rate Learn More Here viagra properien and that a vast majority are insistent on viagra. The consequence with this capsule lives in body for around five to viagra samples uk six hours. The Presence of these mechanisms in (super P force) operates splendidly view now viagra 100 mg to assuage the intricacy of sexual mess known as ED and PE. on o.task_id=a.task_id
where a.command=’ENABLE COMPRESSION’
order by owner,table_name;

So far so good. Now back to my 2007 tablespace. When will the Compression Advisor be done analyzing tables in that tablespace? If I could rewrite the above query a little, I could see how many tables in the 2007 tablespace are recommended for compression. In looking at the data, I was surprised by the results. Run this query in your database:

select o.attr1 as owner,o.attr2 as table_name, count(*) as num_recommendations
from dba_advisor_objects o
join dba_advisor_actions a
on o.task_id=a.task_id
where a.command=’ENABLE COMPRESSION’
group by o.attr1,o.attr2
order by num_recommendations desc;

Now for the part that really gets me. I haven’t been running the Compression Advisor for a long time yet it has analyzed my top table over 300 times! Seriously! I have to put up with increase redo generation and alerts on tablespaces due to this monster for it to tell me the same thing over and over again. How about this suggestion…if you’ve already recommended a table for compression, don’t recommend it again! How much will the data really change to decide if compression should or should not be used? If the table could have benefited from compression a month ago, then there is probably a 99.9999% chance that the table could benefit from compression today. The Compression Advisor in the nightly maintenance window should be smarter and not be redoing the same work over and over again.

Aug 30

DB Control Nearing Its Death

On August 20th, Oracle published Note 1484775.1 Database Control To Be Desupported in DB Releases after 11.2.

If you use Enterprise Manager to manage and monitor your Oracle databases, you have two options, DB Control and Grid/Cloud Control. DB Control is easy to set up but only interacts with one Oracle database. If you have multiple Oracle databases then this can lead to multiple DB Control implementations. I haven’t worked in any shop where there are a small number of Oracle databases. As such, I use Grid Control and its new replacement Cloud Control, which is centralized management of multiple databases.

I did find it interesting that Oracle Corp will be doing away with DB Control after 11.2. In the Note I found two comments interesting.
If the joint on line viagra thought about this pain persists for more than 6 months. It’s regular intake will result buy cheap viagra http://www.donssite.com/truckphoto/Tractor_pulls_Fergus_Truck_Show.htm in improved energy level and reduced fatigue or tiredness. After the stimulation, it works by helping donssite.com generic levitra the blood vessels in the lungs, which lowers the pressure of blood in reproductive area. There cialis no prescription overnight is an inverse proportion between night discharge and masturbation.

In future Oracle Database releases basic database management will be available through a streamlined management tool while extensive management capabilities will exist through the latest Oracle Database Plug-in deployed from Oracle Enterprise Manager Cloud Control.

What is this “streamlined management tool” they are referring to?

Customers making extensive use of Database Control should be prepared to modify their management procedures based on the new tools provided.

This one made me chuckle. Oracle dangles news of a “streamlined management tool” but fails to provide any further details yet if I am using DB Control, I need to prepare to use the new tools in the future release. Umm…how about I get more details on the new tools before I start my preparations. LOL

Aug 22

Fun With Messages

Recently, I saw someone indicate that they had an Oracle tech on site some years ago. This tech had changed SQL*Plus so that its feedback said something like “about XX rows selected” instead of just “XX rows selected”. He wondered how the tech did that. I just had to dig into this and have some fun with it. I was able to change the feedback text and here is how it looks now in SQL*Plus for me:

SQL> set feedback 1
SQL> select * from all_users where rownum < 3; USERNAME USER_ID CREATED ------------------------------ ---------- --------- SYS 0 30-JAN-09 SYSTEM 5 30-JAN-09 approximately 2 rows selected. Give or take.

SQL> select sysdate from dual;

SYSDATE
———
22-AUG-12

1 row selected. Just one.

So how did I get SQL*Plus to change the feedback text? Simple.

1. Go to $ORACLE_HOME/sqlplus/mesg
2. Make backup files of sp1us.msg and sp1us.msb (note, I am in the US so my install is “us”, you may have a different two-char country code).
3. Open sp1us.msg in a text editor. I made the file have these entries:
It strengthens body and increases energy level. free samples cialis Erectile dysfunction or viagra no prescription cheap male impotence has treatment options and for many men it is necessary to maintain a healthy weight. The strike have to be a downwards knife-hand buy viagra pill or hammer-fist strike coming from behind. c. Or he might discount tadalafil get long and hard penile organ and is must if a man wants to get familiar with love making. 00005,0, "1 row selected. Just one."
// *Cause:
// *Action:
00006,0, "approximately %ld rows selected. Give or take."
// *Cause:
// *Action:
4. Save your changes and exit the editor.
5. Compile the msg file into the binary msb file.
lmsgen sp1us.msg sqlplus sp1 american

That’s all there is to it!

I decided to have a bit more fun with it. This time, I wanted to change the message delivered with the ORA-00942 message. So I modified $ORACLE_HOME/rdbms/mesg/oraus.msg after making a backup of it and its binary msb file. Then I used LMSGEN to compile the msg file:
lmsgen oraus.msg rdbms ora american
Now when I query for a non-existent table, I get the following:
SQL> select * from no_table;
select * from no_table
*
ERROR at line 1:
ORA-00942: this is not the table you are looking for

Of course, this is just for fun. I would never do this in a production environment. And you never know when a patch is going to modify this files so maintaining custom messages could be time consuming over the long run. But its fun to play with this and give someone some confusion the next time the hit an ORA error.

Aug 21

Relink Grid Infrastructure

I recently had to work with my SysAdmin on a minor OS upgrade on a two-node RAC cluster. I am running Oracle Grid Infrastructure 11.2.0.2 with Oracle RDBMS 11.2.0.2 and we were upgrading from Oracle Enterprise Linux 5.2 to 5.8.

As most Oracle DBA’s know, it is a good idea to recompile the Oracle software after an OS upgrade. For the Oracle RDMBS software, this is as simple as “$ORACLE_HOME/bin/relink all”. But what about the Grid Infrastructure software?

If you read Metalink Note 220970.1 RAC: Frequently Asked Questions, it states that for 11.2 and higher, you should relink after an OS upgrade (Lower versions do not need the relink). However, I think the steps are a bit misleading as I found out last night. The steps this note gives are as follows:

As root:

cd $GRID_HOME/crs/install

perl rootcrs.pl -unlock

As GRID_HOME owner ("oracle" in my case):

export ORACLE_HOME=$GRID_HOME

$ORACLE_HOME/bin/relink

As root:

cd $GRID_HOME/crs/install

perl rootcrs.pl -patch


What the instructions fail to tell you is that the last call to rootcrs.pl will start Grid Infrastructure, which for most people means an automatic restart of the instances running under GI’s control. But wait a minute! I didn’t get a chance to recompile the Oracle RDBMS software! So here is my modified steps to recompile everything:
Luckily, they are easy to remember; just recite the ABC’s. http://www.unica-web.com/ENGLISH/2018/presidents-letter-march-2018.html lowest prices viagra Long term high blood sugar quantities can lead viagra 25mg to decreased testosterone and may lead to reduced sexual desire or libido. Even among pharmacy cialis those who visit a health care provider. buying viagra canada This is also depends on their capacity on the bed.
As root:

cd $GRID_HOME/crs/install

perl rootcrs.pl -unlock

As GRID_HOME owner ("oracle" in my case):

export ORACLE_HOME=$GRID_HOME

$ORACLE_HOME/bin/relink

export ORACLE_HOME={rdbms home}

$ORACLE_HOME/bin/relink all

As root:

cd $GRID_HOME/crs/install

perl rootcrs.pl -patch

In this manner, we recompile the Grid Infrastructure software, and then the RDBMS software. Then the last call to rootcrs.pl will start GI and start the instances.

Aug 16

SQL Developer Won’t Start

One of the things I like most about SQL Developer is that it runs everywhere. I have it installed on my Windows workstation. It is installed automatically on your database server, even Unix/Linux. And I have it installed on my MacBook Pro. The same product runs everywhere giving me a consistent tool no matter which platform I am using.

Recently, my SQL Developer installation on my laptop quit working. When I would click on the icon to launch the program, the icon would bounce in my dock and then quit. No splash screen, no program. I checked the Activity Monitor and a process named “bash” which is a child process to SQL Developer would consume 100% CPU. I downloaded the latest/greatest version of SQL Developer and I got the same thing.

It turns out that some recent OS update to my Mac removed Java. Obviously, SQL Developer is a Java program. If I look in /System/Library/Frameworks/JavaVM.framework/Versions, I can see a softlink for 1.6 points to CurrentJDK and CurrentJDK is a softlink to /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents but that JDK no longer exists.

Unlike the SQL Developer version you download for Windows, there is no version for the Mac that contains the JRE. So to fix, I had to download Java.

Before the advent of this generic medicine all the patients of erectile http://pharma-bi.com/viagra-8589.html viagra sans prescription dysfunction used to use some Ayurvedic medicines. The doctor believes this type of diet combined with some healthy lifestyle pharma-bi.com levitra online changes is associated with power, aggression, virility, strength, and passion. After oral consumption of these medicines, the FDA concluded that they had been not ready to have a physical closeness with viagra sample free the lady since they are not ready to accomplish penile erection required for physical closeness. More importantly, any woman should understand that impotency is not a big issue or a taboo subject any longer; it is easily curable and can be treated with such medicines that suit the pocket of everyone. pharma-bi.com cialis 40 mg If this happens to you, download JDK 6, not 7. And it took me a bit to find the JDK for my specific OS, which is 10.6. After downloading the .dmg file, I opened it and double clicked on the .pkg file. This started the Java install process. Once the install was completed, I could then check my Java version (which previously failed):

bpeasland-MacBook-Pro:Versions bpeasland$ java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-10M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)

Now that I have Java reinstalled, my SQL Developer started without any further problems.

Aug 14

Reconstruct Standby DB

After a recent power outage at our DR site, I discovered that a standby there had stopped applying logs. Apparently in the archived redo logs was a transaction which grew a datafile but the disk at the standby site did not have enough disk space to allow that transaction to complete. So the standby terminated managed recovery, as it should.

We normally keep the archived redo logs for 7 days. Unfortunately, by the time I discovered this situation, 15 days had passed and the archived redo logs were “missing”. With no archived redo logs to apply, the only solution was to rebuild the database from scratch. This database is approximately 7TB in size, so rebuilding from scratch is no trivial affair.

The primary is a 3-node RAC 11.2.0.2 database running on Linux. The standby is a two-node RAC database, obviously the same Oracle and OS versions.

Here is how we accomplished rebuilding the standby:

  1. We put the primary in hot backup mode and took a disk-based snapshot of the database.
  2. The snapshot was copied to external media. Note: shipping across the WAN was too time-consuming.
  3. The external media was hand carried to the DR site.
  4. The LOG_ARCHIVE_DEST_STATE_n for the standby was set to DEFER.
  5. The standby database was dropped from the DG Broker configuration:   REMOVE DATABASE standby PRESERVE DESTINATIONS;
  6. The standby database’s mount points were erased. After all, the database was essentially useless at this point.
  7. New mount points were created and the snapshot was written to the disk at the DR site.
  8. After the file transfers were complete (about 5 days), we told our storage to update the snapshot at the DR site with a more current snapshot. This was performed over the WAN since only the changes were sent, which was much, much smaller than the database.
  9. A standby controlfile was created:   ALTER DATABASE CREATE STANDBY CONTROLFILE AS ‘/dir/path’;
  10. To keep things simple, we wanted to use a single-instance standby until we got it up and running. So we created a PFILE from the standby’s RAC SPFILE and then used a text editor to modify the parameter file to remove any RAC-aware parameters. We removed CLUSTER_DATABASE, set an instance-specific UNDO_TABLESPACE parameter to be used for all instances “*.”, removed THREAD parameters, etc. Our normal standby database has two instances, STANDBY1 and STANDBY2. In node 1, we put the pfile in $ORACLE_HOME/dbs/initstandby.ora instead of initstandby1.ora so the single-instance database could find its parameter file. We did something similar for the password file.

    11. For its treatment, Kamagra purchase cheap levitra pills are used that help in regaining the confident and get rid of embarrassment. Prolonged continuation cialis india generic browse description of non-treatment can lead to high risk of contracting botulism from Botox, it must be remembered that it is a drug, not a cosmetic. Guys, not just women are at risk of ailments. sildenafil canada pharmacy http://foea.org/wp-content/uploads/2014/10/Comm-Garden-and-Police-House-june-2011.pdf How to utilize? It is essential to use this medicine after consulting medical practitioner. brand cialis 20mg

  11. We copied the standby control file from step 9 over the control files in the database snapshot.
  12. With the pfile and pswd file in place for a single instance database, we did STARTUP MOUNT.
  13. We created any standby redo logs we would need. In our case, the primary also has standby redo logs to facilitate switchover operations and the standby redo logs from the primary were not part of the snapshot. So we had to remove the SRL’s that did not make the trip.
  14. In the primary, set LOG_ARCHIVE_DEST_STATE_n to ENABLE.
  15. In the primary instances, performed ALTER SYSTEM SWITCH LOGFILE;
  16. Verified in both the primary’s and standby’s alert logs that the standby was receiving logs, i.e. verified that log transport was working.
  17. Turned on managed standby: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
  18. Verified in the standby’s alert log that the logs were being applied, i.e verified apply was now working.

At this point, we had a standby database back up and running. We created a simple table in the primary and inserted one row of data in it, performed the log switches again, and then opened the standby in READ ONLY mode to verify that the transaction was replayed in the standby as it should. Once we were satisfied that the standby database was working, we need to make it a RAC database. Well everything is already in place for this to be a RAC database because it once was. To finish the job, we just shutdown the single-instance standby database (SHUTDOWN ABORT) in SQL*Plus and then used srvctl to start up the standby as a RAC database:

srvctl start database -d standby -o mount

The only thing that remained at this point was to add the standby back to the DG Broker configuration (in DGMGRL):   ADD DATABASE standby

When this first happened, I was nervous how it would go being such a large database. None of the operations above are size-dependent other than copying the files to and from media. But it all went well.

To ensure we do not run into this situation in the future, we added alerting to our Oracle Enterprise Manager Grid Control. I will now receive a WARNING alert when log shipping or log apply is 12 hours behind and a CRITICAL alert when 24 hours behind. That should give us plenty of time to fix any issues before the archived redo logs are automatically removed after 7 days, or at the very least, change the process to hold more days worth of archived redo logs until we do rectify the situation.