ORA-27300: OS system dependent operation:fork failed with status: 11

I have encounter below error on all the databases on a shared server.

Errors from alert_<SID>.log

Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file

I see that m000 process dies and oracle has problem forking more processes.

Trace file ORA1_ psp0_18712.trc contents

*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780

Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)

*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….

*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5

It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue

For Linux
Increase the nproc in /etc/security/limits.conf file

For Solaris 10
Increase the project.max-lwps value

For HP and Solaris
Increase the maxuprc

For AIX
Increase maxuproc

I didn’t see the issue or error in the alert.log file.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com