Terminating the instance due to error 16038 and unable to startup

I have encountered storage crash on one of the Database system. After storage is back online, not able to bring up the database. The database comes to mount stage and terminating the instance.

We are receiving below error.

ORA-03113: end-of-file on communication channel
Process ID: 5038
Session ID: 202 Serial number: 3

When I look into the alert.log I found below error messages

ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Tue Mar 10 10:24:10 2015
Incomplete read from log member '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'. Trying next member.
ARCH: All Archive destinations made inactive due to error 333
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle/ORAC/archiveORAC_ARCH_347548987.dbf' (error 333) (ORAC)
Committing creation of archivelog '/oracle/ORAC/archive/1_3541_854889367.dbf' (error 333)
Errors in file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_ora_23498.trc:
ORA-16038: log 1 sequence# 3541 cannot be archived
ORA-00333: redo log read error block count
ORA-00312: online log 1 thread 1: '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'
USER (ospid: 23498): terminating the instance due to error 16038
Tue Mar 10 10:24:12 2015
System state dump requested by (instance=1, osid=23498), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_diag_23470.trc
Dumping diagnostic data in directory=[cdmp_20150310102412], requested by (instance=1, osid=23498), summary=[abnomal instance termination].
Instance terminated by USER, pid = 23498

By looking the error, I can see that one of the online redo is corrupted. Archive process cannot able to archive to disk. In order to bring up the database you need to clear the corrupted redo log.

From ALERT.LOG file, I am able to identify that group 1 redo log 1 corrupted. By using below command I have cleared the unarchived log file

SQL> alter database clear unarchived logfile ‘/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

If you find all multiplexed redo members are corrupted then you need to run below command to clear the GROUP.

SQL> alter database clear unarchived logfile group 1;

Now I am able to bring up the database. But note that you cannot able to bring the database back beyond corrupted point using old backups. Now you must take one FULL database backup.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
http://www.sqlserver-expert.com