When trying to start-up a database received below error on a REHL 5.x
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
By looking the error, you might think that it is a space issue. But note that the actual error is “Out of Memory”.
You can also check available memory on the server and you should see enough FREE memory.
$free -g
total used free shared buffers cached
Mem: 125 80 45 0 0 88
-/+ buffers/cache: 36 89
Swap: 23 2 21
The server has enough memory but out of shared memory pages that can be used system wide. You should increase the kernel.shmall value to fix the issue.
The default size for SHMALL in Red Hat Enterprise Linux 2.1, 3, 4 and 5 is 2097152. Refer My Oracle Support document ID 169706.1 for recommended values.
Once you change the value you no need to reboot the server, you should reload the system configuration file as root using below command
$ sysctl –p
Hope this should resolve your issue
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Friday, December 19, 2014
Tuesday, December 2, 2014
Recovery Manager(RMAN) New Features in Oracle Database 12c - Part2
Hi Everyone,
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
Oracle Database 12c – RMAN New Features: Part2
In this article I will cover:
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
Oracle Database 12c – RMAN New Features: Part2
In this article I will cover:
- Multisection Backup Improvements
- Restoring and Recovering Files Over Network
- Storage Snapshot Optimization
- Active Database Duplication Improvements
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
ORA-27300: OS system dependent operation:fork failed with status: 11
I have encounter below error on all the databases on a shared server.
Errors from alert_<SID>.log
Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file
I see that m000 process dies and oracle has problem forking more processes.
Trace file ORA1_ psp0_18712.trc contents
*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780
Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)
*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….
*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue
For Linux
Increase the nproc in /etc/security/limits.conf file
For Solaris 10
Increase the project.max-lwps value
For HP and Solaris
Increase the maxuprc
For AIX
Increase maxuproc
I didn’t see the issue or error in the alert.log file.
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Errors from alert_<SID>.log
Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file
I see that m000 process dies and oracle has problem forking more processes.
Trace file ORA1_ psp0_18712.trc contents
*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780
Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)
*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….
*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue
For Linux
Increase the nproc in /etc/security/limits.conf file
For Solaris 10
Increase the project.max-lwps value
For HP and Solaris
Increase the maxuprc
For AIX
Increase maxuproc
I didn’t see the issue or error in the alert.log file.
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Friday, November 14, 2014
Oracle GoldenGate 12c New Features - Part1
Hi Everyone,
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please see the article using below link.
Oracle Golden Gate 12c New Features – Part 1
In this article I will cover:
- Expanded heterogeneous Support
- Multitenant Container Database (CDB) Support
- Oracle Universal Installer (OUI) Support
- Support for Public and Private Clouds
- Integrated Replicat
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)
User may encounter below error in 11g. First I would recommend running ADDM report and checking how the resources are utilized. If you see any undersize database configuration then increase.
ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)
Incident details in: /u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
Writing to the above trace file is disabled for now on...
If your database has enough resources and if you still see the issue then you might need to check the trace file.
=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
4129 MB total:
4129 MB commented, 173 KB permanent
57 KB free (0 KB in empty extents),
2240 MB, 3 heaps: "koh-kghu call " 14 KB free held
1889 MB, 1 heap: "session heap " 4 KB free held
------------------------------------------------------
The trace file confirms that the process is limiting at 4GB. Change the upper limit either at OS level or Database level
Apply this change at OS level
1. Get the page count using below command and increase the limit
#more /proc/sys/vm/max_map_count
2. Increase the page count
#sysctl -w vm.max_map_count= 262144
or
#sysctl vm.max_map_count=262144
3. These changes will be revered on server reboot. To make changes permanent add the value in
$ vi /etc/sysctl.conf
or
$echo “vm.max_map_count = 262144” | tee -a /etc/sysctl.conf
4. You no need to reboot the server, you should reload the config file as root
$ sysctl –p
Apply this change at Database level
Adjust the realfree heap page size by setting below parameters in spfile or pfile and restart the database
_use_realfree_heap=TRUE
_realfree_heap_pagesize_hint = 262144
Note that the default realfree allocator pagesize is 64 kB (65536), so 64K entries take up 4GB. With 256kB (262144) pages, the limit goes to 16GB.
I hope this should resolve your issue.
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)
Incident details in: /u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
Writing to the above trace file is disabled for now on...
If your database has enough resources and if you still see the issue then you might need to check the trace file.
=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
4129 MB total:
4129 MB commented, 173 KB permanent
57 KB free (0 KB in empty extents),
2240 MB, 3 heaps: "koh-kghu call " 14 KB free held
1889 MB, 1 heap: "session heap " 4 KB free held
------------------------------------------------------
The trace file confirms that the process is limiting at 4GB. Change the upper limit either at OS level or Database level
Apply this change at OS level
1. Get the page count using below command and increase the limit
#more /proc/sys/vm/max_map_count
2. Increase the page count
#sysctl -w vm.max_map_count= 262144
or
#sysctl vm.max_map_count=262144
3. These changes will be revered on server reboot. To make changes permanent add the value in
$ vi /etc/sysctl.conf
or
$echo “vm.max_map_count = 262144” | tee -a /etc/sysctl.conf
4. You no need to reboot the server, you should reload the config file as root
$ sysctl –p
Apply this change at Database level
Adjust the realfree heap page size by setting below parameters in spfile or pfile and restart the database
_use_realfree_heap=TRUE
_realfree_heap_pagesize_hint = 262144
Note that the default realfree allocator pagesize is 64 kB (65536), so 64K entries take up 4GB. With 256kB (262144) pages, the limit goes to 16GB.
I hope this should resolve your issue.
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Thursday, November 13, 2014
SPFILE restore failed with RMAN-04014 and RMAN-04014
User normally restores SPFILE when it is deleted accidently or the database is completed decommissioned and database need to restore from backup.
To restore SPFILE user must set the DBID
RMAN> set DBID=4563434343232
Before you restore SPFILE you need to bring up the database in nomount. But there is not SPFILE to bring the database.
Where SPFILE is not available startup the database with “STARTUP FORCE NOMOUNT”, RMAN will start the instance with a dummy parameter
You might see below error when trying to startup the database
RMAN> startup force nomount;
executing command: SET DBID
database name is “TST” and DBID is 4563434343232
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file ‘/u01/home/oracle/product/10.2.0/db_1/dbs/initTST.ora’
starting Oracle instance without parameter file for retrival of spfile RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 11/13/2014 15:37:45
RMAN-04014: startup failed: ORA-04031: unable to allocate 4128 bytes of shared memory (“shared pool”,”unknown object”,”sga heap(1,0)”,”kglsim hash table bkts”)
As you see ERROR message in bold, the default values to startup the database is not sufficient. These default values depend up on the version and operating system.
Set the SGA_TARGET environment variable with higher value at O/S level.
export ORA_RMAN_SGA_TARGET= 10240
RMAN> startup force nomount;
startup failed: ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/u01/home/oracle/product/10.2.0/db_1/dbs/initTST.ora'
starting Oracle instance without parameter file for retrival of spfile
Oracle instance started
Total System Global Area 107374182400 bytes
Fixed Size 2286808 bytes
Variable Size 1325402920 bytes
Database Buffers 106032005120 bytes
Redo Buffers 14487552 bytes
Now the database instance opened NOMOUNT state and you can restore the SPFILE
RMAN> restore spfile from autobackup;
Starting restore at 2014-11-13:15:40:35
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=33 devtype=DISK
channel ORA_DISK_1: looking for autobackup on day: 20141113
channel ORA_DISK_1: looking for autobackup on day: 20141112
channel ORA_DISK_1: looking for autobackup on day: 20141111
channel ORA_DISK_1: looking for autobackup on day: 20141110
channel ORA_DISK_1: looking for autobackup on day: 20141109
channel ORA_DISK_1: looking for autobackup on day: 20141108
channel ORA_DISK_1: looking for autobackup on day: 20141107
channel ORA_DISK_1: no autobackup in 7 days found
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 11/13/2014 15:40:37
RMAN-06172: no autobackup found or specified handle is not a valid copy or piece
The SPFILE restore has failed because the database mounted with DUMMY spfile and it does not have information about AUTOBACKUP of the SPFILE.
You have two options.
1. Restore the SPFILE using FROM clause
RMAN> restore spfile from '/ora-backup/TST/cf_c-2786659778-20141110-05';
Starting restore at 2014-11-13:15:40:59
using channel ORA_DISK_1
channel ORA_DISK_1: autobackup found: /ora-backup/TST/cf_c-2786659778-20141110-05
channel ORA_DISK_1: SPFILE restore from autobackup complete
Finished restore at 2014-11-13:15:41:02
2. Restore the SPFILE using Control file AUTOBACKUP
RMAN> RUN {
SET CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO 'autobackup_format';
RESTORE SPFILE FROM AUTOBACKUP;
}
executing command: SET CONTROLFILE AUTOBACKUP FORMAT Starting restore at 13-JUN-13
using channel ORA_DISK_1 channel ORA_DISK_1: looking for AUTOBACKUP on day: 20141113
channel ORA_DISK_1: AUTOBACKUP found: '/ora-backup/TST/cf_c-2786659778-20141110-05’
channel ORA_DISK_1: restoring spfile from AUTOBACKUP '/ora-backup/TST/cf_c-2786659778-20141110-05
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete Finished restore at 2014-11-13:16:49:02
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Tuesday, October 7, 2014
Recovery Manager New Features in Oracle Database 12c - Part1
Hi Everyone,
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
Oracle Database 12c – RMAN New Features: Part1
In this article I will cover:
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
Oracle Database 12c – RMAN New Features: Part1
In this article I will cover:
- SQL Interface Improvements
- SYSBACKUP Privilege
- Support for multitenant container and pluggable databases
- DUPLICATE enhancements
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Monday, September 8, 2014
Article: Should you install or clone Oracle Home?
Hi Everyone,
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
This article will help all Oracle Community to understand when cloning is useful, available methods and how to clone an Oracle Installation.
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
Should you install or clone Oracle Home?
In this article I have covered- When is cloning useful
- Methods available for cloning
- How is Cloning done?
This article will help all Oracle Community to understand when cloning is useful, available methods and how to clone an Oracle Installation.
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Communication error with server. Callback failed for data entry page.
I have recently encountered below error when working with Hyperion planning.
Communication error with server. Callback failed for data entry page.
Cause:
There was a problem in the software code in the jsp file: EnterData_Inner.jsp which is executed when opening a Planning web form.
Steps to reproduce the error
The issue can be reproduced at will with the following steps:
1. Log into Workspace and open a Planning application.
2. Open a form.
3. Get error above.
2. <Add step details>
Proposed Solution or workaround.
During the troubleshooting found that this was a Bug and it is fixed in “Patch Set Update: 11.1.2.1.102 Oracle Hyperion Planning, Fusion Edition” or “patch 14126387”.
Also issue this is fixed in PSU .600 patch, but you will have to apply patches other products as per-requisites.
Bug 13588551 - "COMMUNICATION ERROR WITH SERVER" ERROR WHILE OPENING DATA FORMS -11.1.2.1.101
Before you start workspace make sure you DELETE all cookies in your browser.
Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Oracle RAC New features in 11gR2 (11.2.0.4)
Oracle 11g R2 (11.2.0.4) has new administration and deployment features for Oracle Real Application Clusters (Oracle RAC).
1. Oracle RAC Configuration Audit Tool
The Oracle RAC Configuration Audit Tool (RACcheck) is designed to audit various important configuration settings within an Oracle RAC system. This tool evaluates single instance and Oracle RAC database installations for best practices, configuration issues, regular health checks, and pre- and post-upgrade best practices.
You can use RACcheck to audit configuration settings for
The Oracle Trace File Analyzer (TFA) Collector is a diagnostic utility, will be useful to simplify diagnostic data collection for Oracle Grid Infrastructure, Clusterware, and Oracle RAC systems.
TFA is installed into the Oracle Grid Infrastructure home when you install, or upgrade to 11.2.0.4. It discovers relevant trace file directories and analyzes the files. The TAF information is stored in a Berkeley database in the Grid home
You can use TFA either on demand and automatically.
Starts the TFA daemon
/etc/init.d/init.tfa start
Stops the TFA daemon
/etc/init.d/init.tfa stop
Stops and then starts the TFA daemon
/etc/init.d/init.tfa restart
Stops the TFA daemon and removes entries from the appropriate O/S configuration
/etc/init.d/init.tfa shutdown
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
1. Oracle RAC Configuration Audit Tool
The Oracle RAC Configuration Audit Tool (RACcheck) is designed to audit various important configuration settings within an Oracle RAC system. This tool evaluates single instance and Oracle RAC database installations for best practices, configuration issues, regular health checks, and pre- and post-upgrade best practices.
You can use RACcheck to audit configuration settings for
- Database parameters and configuration settings important to Oracle RAC
- Operating system kernel parameters
- Oracle Grid Infrastructure
- Operating system packages and configuration
- Upgrade readiness assessment to Oracle Database 11g release 2 (11.2)
- Oracle Database and Oracle ASM (Oracle Automatic Storage Management
- Intel Linux (Enterprise Linux, RedHat and SuSE 9,10, 11)
- Oracle Solaris SPARC (Solaris 10 and 11), x86-64 (Solaris 10 and 11)
- IBM AIX
- HP-UX
The Oracle Trace File Analyzer (TFA) Collector is a diagnostic utility, will be useful to simplify diagnostic data collection for Oracle Grid Infrastructure, Clusterware, and Oracle RAC systems.
TFA is installed into the Oracle Grid Infrastructure home when you install, or upgrade to 11.2.0.4. It discovers relevant trace file directories and analyzes the files. The TAF information is stored in a Berkeley database in the Grid home
You can use TFA either on demand and automatically.
- Use tfactl set to enable automatic collection of diagnostics
- Use tfactl diagcollect command to collect trimmed trace files for any component and time range
Starts the TFA daemon
/etc/init.d/init.tfa start
Stops the TFA daemon
/etc/init.d/init.tfa stop
Stops and then starts the TFA daemon
/etc/init.d/init.tfa restart
Stops the TFA daemon and removes entries from the appropriate O/S configuration
/etc/init.d/init.tfa shutdown
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Thursday, July 3, 2014
Oracle 12c - One Database and One Solution
Hi Everyone,
My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate
Please view the article using below link.
Oracle 12c – One Database and One Solution
In this article i have covered
- What is Multitenant Architecture?
- The benefits of managing many database as ONE database
- Why Oracle 12c?
- Upgrade path to Oracle 12c
- Oracle Enterprise Manage Cloud Control
This article helps all Oracle community who don’t understand what Oracle 12c offers and how they can reduce IT Costs.
I hope you will like the article and it will be helpful to you.
Please leave your valuable comments.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Tuesday, June 24, 2014
java.sql.SQLRecoverableException: IO Error: Connection reset
I have come across below error during Oracle Agile Managed Server deployment (Agile 9.3.3). When trying to startup the Managed server which is using Weblogic12c, it didn’t startup and found below error in logfile.
Your environment has been set.
AgileAuthenticationProviderImpl.initialize
log4j:WARN No appenders could be found for logger (com.agile.util.sql.OracleConnectionImpl).
log4j:WARN Please initialize the log4j system properly.
java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:467)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:207) at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPool Data Source.java:139) at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection (Oracle ConnectionPool Data Source.java:88) at........ com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method) at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
at com.sun.proxy.$Proxy13.login(Unknown Source) at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89) at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
at com.sun.proxy.$Proxy34.authenticate(Unknown Source) at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40) at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.
java:930) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl. initialize(CommonSecurityServiceManagerDelegateImpl.java:1054) at
weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at oracle.net.ns.DataPacket.send(DataPacket.java:210)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:230)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:312)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket (T4CSocketInputStreamWrapper. java:124) at oracle.jdbc.driver.T4CSocketInputStream Wrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:404)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:385)
... 59 more
java.sql.SQLRecoverableException: IO Error: Connection reset
You will also see below error in the log file, but these errors will mislead you
<Jun 19, 2014 2:13:44 PM PDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>
<Jun 19, 2014 2:13:44 PM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
...(CommonSecurityServiceManagerDelegateImpl.java:960) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054) at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.LoginException: java.lang.ExceptionInInitializerError
at com.agile.admin.security.weblogic.WLSLoginModule.login
(WLSLoginModule.java:193) at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper. java: 110 at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper .java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Findings/troubleshooting
1. When trying to startup the Managed server the weblogic taking more time than expected
2. If you read the highlight error then you will understand what the exact cause for this error is “java.sql.SQLRecoverableException: IO Error: Connection reset”.
3. Found below warning in alert<sid>.log
WARNING: inbound connection timed out (ORA-3136)
4. The userid/password is correct
By looking the above findings it seems that weblogic taking more time and it was not able to connect to the database and timed out with ORA-3136 error. It looks like inbound connection timeout occurred.
Proposed Solution: I have changed the SQLNET.INBOUND_CONNECT_TIMEOUT from default (60 sec) value to 0(indefinite).
After changing the parameter the weblogic Instance comes online without any issues.
It looks like there is bug in Weblogic due to which it is not able to connect to database with in the specified time. Still troubleshooting to find the root cause, if I find anything will try to update in the post.
Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
Subscribe to:
Posts (Atom)
