ORA-27102: out of memory Linux-x86_64 Error: 28: No space left on device

When trying to start-up a database received below error on a REHL 5.x

ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device 

By looking the error, you might think that it is a space issue. But note that the actual error is “Out of Memory”.

You can also check available memory on the server and you should see enough FREE memory.

$free -g
total used free shared buffers cached
Mem: 125 80 45 0 0 88
-/+ buffers/cache: 36 89
Swap: 23 2 21

The server has enough memory but out of shared memory pages that can be used system wide. You should increase the kernel.shmall value to fix the issue.

The default size for SHMALL in Red Hat Enterprise Linux 2.1, 3, 4 and 5 is 2097152. Refer My Oracle Support document ID 169706.1 for recommended values.

Once you change the value you no need to reboot the server, you should reload the system configuration file as root using below command

$ sysctl –p

Hope this should resolve your issue

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Recovery Manager(RMAN) New Features in Oracle Database 12c - Part2

Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please view the article using below link.

Oracle Database 12c – RMAN New Features: Part2

In this article I will cover:

• Multisection Backup Improvements
• Restoring and Recovering Files Over Network
• Storage Snapshot Optimization
• Active Database Duplication Improvements

This article will help all Oracle Community to understand new enhancements and additions in Recovery Manager (RMAN) and take advantage of new features for efficient backup & recovery.

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-27300: OS system dependent operation:fork failed with status: 11

I have encounter below error on all the databases on a shared server.

Errors from alert_<SID>.log

Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file

I see that m000 process dies and oracle has problem forking more processes.

Trace file ORA1_ psp0_18712.trc contents

*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780

Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)

*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….

*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5

It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue

For Linux
Increase the nproc in /etc/security/limits.conf file

For Solaris 10
Increase the project.max-lwps value

For HP and Solaris
Increase the maxuprc

For AIX
Increase maxuproc

I didn’t see the issue or error in the alert.log file.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com