Copyright 2013 - http://www. Weblogic-Wonders.com

We were observing frequent server hangs.
On taking thread dumps at the time of hang we found the following culprit thread.

[STUCK] ExecuteThread: ’6′ for queue: ‘weblogic.kernel.Default (self-tuning)’” daemon prio=1 tid=01baae30 nid=55 lwp_id=9965842 runnable [1ff05000..1ff03878]
at oracle.jdbc.oci8.OCIDBAccess.do_fetch(Native Method)
at oracle.jdbc.oci8.OCIDBAccess.executeFetch(OCIDBAccess.java:1758)
- locked < 36582dd0> (a oracle.jdbc.oci8.OCIDBAccess)
at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2659)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2832)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:608)
- locked < ae1c1400> (a oracle.jdbc.driver.OraclePreparedStatement)
- locked < 36589900> (a oracle.jdbc.driver.OracleConnection)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:536)
- locked < ae1c1400> (a oracle.jdbc.driver.OraclePreparedStatement)
- locked < 36589900> (a oracle.jdbc.driver.OracleConnection)
at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:100)

We solved the issue by switching to Type 4 driver.

The post Weblogic Server Stuck thread and very high cpu usage when executing a query appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

Below are few most common issues you might run into when trying to start a NodeManager.

1.

weblogic.nodemanager.common.ConfigException: Native version is enabled but NodeManager native library could not be loaded

Caused by: java.lang.UnsatisfiedLinkError: no nodemanager in java.library.path

Remedy:

You would need to point your jmv to point to the native library nodemanager.dll file.

This can be done in two ways.

set LD_LIBRARY_PATH=%BEA_HOME%\wlserver\serve\native\win\32

set JAVA_OPTIONS=%JAVA_OPTIONS%; %LD_LIBRARY_PATH%;

or

set JAVA_OPTIONS=%JAVA_OPTIONS%;  -Djava.library.path=”C:\Oracle\Middleware\Oracle_Home\wlserver\server\native\win\32″

2.

Caused by: java.lang.UnsatisfiedLinkError: C:\Oracle\Middleware\Oracle_Home\wlserver\server\native\win\32\nodemanager.dll: Can’t load IA 32-bit .dll on a AMD 64 -bit platform

Remedy:   

1.  Create a nodemanager folder under   <BEA_HOME>\ common folder if it’s a fresh installation as by default this folder is not created

2. Create a nodemanager.properties file under the created nodemanager folder and set the below property.

 NativeVersionEnabled=false

3. Start the NodeManager using the startNodeManager script.

The post Common NodeManager exceptions (UnsatisfiedLinkError) appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

You need to change the web-app version to 2.5  as previous version does not support JSP 2.1. Weblogic server enforces the jsp version mentioned in the web-app of the web.xml.

It should be like this.

<web-app
xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
version="2.5"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd">

I have uploaded a sample application which can be dowloaded from here

The post trimDirectiveWhitespaces not working? appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

javax.servlet.ServletException: java.lang.NoClassDefFoundError:

Application was failing to find an application class.
Using jarscan we searched the jars containing this class from the middleware home.
The jars were present in the domain lib.
We also checked the classpath in the startWeblogic.cmd and setDomainEnv.cmd to check if direct references were present in the server classpath.

The we logged in to Weblogic Console and analyzed the classloading conflict from the classloader tool.

You can find more details here.

http://docs.oracle.com/cd/E24329_01/web.1211/e24368/classloading.htm#BABHJGJJ

We kept the jar containing the class in the APP-INF lib of the EAR and added the following in the weblogic-application.xml

<prefer-application-packages>
        <package-name>com.demo.*</package-name>
</prefer-application-packages>

Using prefer-application-packages we can ensure that our application packages are loaded and not Weblogic Server’s.

After updating the application the issue was resolved.

The post Solving NoClassDefFoundError using the classloader analysis tool (CAT) appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

The post Memory Leak Overview appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

[E] [ServiceStart] Error in JVM. Cause=Unable to find a JVM!

If we look at the complete log file we will observe the following

lpszCmdLine = -client -Xms32m -Xmx200m -XX:MaxPermSize=128m -XX:+UseSpinning -Dweblogic.Stdout=C:\Oracle\Middleware\user_projects\domains\ClassicDomain\stdout.txt -

The service is starting Weblogic Server with -client option.

-server option should be used on a Windows 64 bit system.

We can either edit the registry and edit the CmdLine value data from -client to -server.
Or we can delete the existing service and create a new service again after modifying the installSvc.cmd file with
set JAVA_VM=-server or for JRockit set JAVA_VM=-jrockit.

The post Issues while starting Managed Server as a Window’s Service appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

<BEA-001112> <Test “SELECT 1 FROM DUAL” set up for pool “Test” failed with exception: “java.sql.SQLException: Io exception: Connection reset by peer: socket write error”.>

Network or the DBMS is killing the connection, you need to check with the Database Admin and the network Admin.

java.lang.ClassCastException: oracle.jdbc.driver.LogicalConnection incompatible with oracle.jdbc.OracleCo
nnection

Check if you’ve got multiple versions of the driver in your environment, server\lib, domain\lib or added in the server classpath in the startWeblogic.cmd. Keep only one driver in your environment.

ORA-28267: Invalid NameSpace value

This is fixed by Patch 13923804 – 11.2.0.2.7 Patch Set Update (11G database patch)

weblogic.common.ResourceException: IO Error: Connection reset atweblogic.jdbc.common.internal.XAConnectionEnvFactory.makeConnection(XAConnectionEnvFactory.java:512)

Try to set the initial capacity of the datasource to zero.

Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource ‘Operational’: XAER_RMERR : A resource manager error has occured in the transaction branch

Issue can occur when you create tables using 10g client on 11g database. Recreate the tables with 11g client.

The post common database related issues with Weblogic Server appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

I/O error while reading domain directory: java.io.FileNotFoundException: Domain directory ‘D:\bea\wlserver_10.3\common\nodemanager’ invalid (domain salt file not found

We need to to nmEnroll and restart the nodemanager to fix this error.

weblogic.nodemanager.common.ConfigException: Native version is enabled but node manager native library could not be loaded

You need to set the LD_LIBRARY_PATH for Solaris/Linux and SHLIB_PATH for HP-UX

Solaris/Linux

LD_LIBRARY_PATH:$WL_HOME/server/lib/solaris:$WL_HOME/server/lib/solaris/ocixxx_x
HP-UX
SHLIB_PATH=$SHLIB_PATH:$WL_HOME/server/lib/hpux11:$WL_HOME/server/lib/hpux11/ocixxx_x

The post NodeManager issues during oracle weblogic server environment setup appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

javax.xml.stream.XMLStreamException: ParseError at [row,col]:[2,134]
Message: Tried all: ’1′ addresses, but could not connect over HTTP to server: ‘java.sun.com’, port: ’80′
at com.sun.xml.stream.XMLReaderImpl.next(XMLReaderImpl.java:545)
at weblogic.servlet.internal.TldCacheHelper$TldIOHelper.parseXML(TldCacheHelper.java:132)
at weblogic.descriptor.DescriptorCache.parseXML(DescriptorCache.java:380)
at weblogic.servlet.internal.TldCacheHelper.parseTagLibraries(TldCacheHelper.java:65)
at weblogic.servlet.internal.War.getTagInfo(War.java:889)
at weblogic.servlet.internal.WebAppServletContext$ServletContextWebAppHelper.getTldInfo(WebAppServletContext.java:3708)

You might encounter the following exception when the server on which Weblogic is hosted is not able to communicate to internet.The reason being that internet access is allowed only via proxy.
To allow access we need to pass on the proxy details as a JVM Option.

-Dhttp.proxyHost=< proxy host> -Dhttp.proxyPort=

This should resolve the issue.

You can test the connection by a simple jsp that makes a URL Connection

System.setProperty(“http.proxyHost”, “proxy host”);
System.setProperty(“http.proxyPort”, “proxy port”);
URL url = new URL(“http://java.sun.com:80″);

The post javax.xml.stream.XMLStreamException while validating xml appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

If you observe CPU being hogged by the Weblogic Server process on a linux machine,
you need to find out the Weblogic Server process id using

ps -ef | grep java

After you’ve found out the process id, find the thread id’s ( light weight process id’s) and their CPU usage using top command.

top -H -p 2389 (Weblogic Server Process Id)

2703 , 2704 & 2705 are the light weight process that are hogging up the CPU.

We take thread dump using kill -3 2389 (Weblogic Server Process Id)

We can map the light weight process to Weblogic Server threads by converting 2703 to hex 0xa8f.

We search for the thread (nid=0xa8f) in the thread dump to know what the thread is doing.
From the stack we can make out some complex mathematical operation is going on.
Inorder to avoid the High CPU we need to change the code to simplify the operation.

“Thread-19″ daemon prio=10 tid=0x8a092000 nid=0xa8f runnable [0x887fe000]
java.lang.Thread.State: RUNNABLE
at java.lang.StrictMath.pow(Native Method)
at java.lang.Math.pow(Math.java:609)
at jsp_servlet.__index$1.run(__index.java:85)
at java.lang.Thread.run(Thread.java:662)

The post High CPU Usage of WebLogic Server on Linux appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

When we try to start the managed server from nodemanager, we get the following error.

<BEA-090064> <The DeployableAuthorizer “myrealm_weblogic.security.providers.xacml.authorization.XACMLAuthorizationProviderImpl” returned an error: weblogic.security.spi.ResourceCreationException: [Security:090310]Failed to create resource.>

This errror comes because the managed server ldap server (slave) is not in sync with the Admin Server ldap server ( master).

To overcome this issue we can use two very important configurations.

Refresh Replica at Startup and Master First.

Master First ensures managed server always connect to the ldap server of the admin server.

Refresh Replica at startup ensures that whenever managed server is restarted, it takes the latest ldap data from the admin server.

The post Unable to start managed server from nodemanager appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

In our live environment we observed very HIGH CPU utilisation on the boxes. Weblogic JAVA process was hogging the CPU.

To find the culprit thread, we followed the steps outlined here.

The below thread was the culprit thread

“C2 CompilerThread1″ #7 daemon prio=9 os_prio=0 tid=0x00002acb10d48000 nid=0x511 runnable [0x0000000000000000]

There were known code cache issues with C2 Compiler Thread so enabled Native Memory Tracking by adding the flag -XX:NativeMemoryTracking=detail to capture native memory usage ( especially code cache.

jcmd [JVM pid] VM.native_memory > VM.native_memory.txt
jcmd [JVM pid] GC.class_histogram > histogram.txt
pmap -x [JVM pid] > pmap.txt

We could see a gradual increase in code cache size.

We monitored the peak usage using JMX and it was around 730 MB.

We turned off tiered compilation (-XX:-TieredCompilation) to reduce code cache usage and this helped stabilize the production environment by reducing CPU usage.

The post High CPU Usage due to C2 CompilerThread appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

Applies To:

Product:

• Oracle WebLogic Server – All Versions

Never ran an RDA (Remote Diagnostic Agent) before? It is a way to collect comprehensive diagnostic information to provide to Oracle Support.

Some Oracle Fusion Middleware installations already have RDA installed under the ~/oracle_common or ~/utils folder. If not or if you have problems running the installed RDA, you may need to download, install, and run it on your own.

Instructions

1. Download RDA from the My Oracle Support via Oracle Patch 21769913.

2. Depending on your operating system and the latest version of RDA, the filename may be called something like p21769913_1931982_Linux-x86-64.zip.

3. Extract the RDA software:

cd /u01/app/oracle/middleware
unzip /tmp/p21769913_1931982_Linux-x86-64.zip
mv readme.txt rda
cd rda

4. Running the RDA for SOA Suite:

./rda.sh -s soa_issue -p OFM_SoaMax

--(follow the prompts)
--(enter /u01/app/oracle/middleware for middleware home)
--(enter /u01/app/oracle/middleware/soa for product oracle home)

gtar -czvf soa_issue.tgz soa_issue

5. Running the RDA for OSB:

./rda.sh -s osb_issue -p OFM_OsbMax

--(follow the prompts)
--(enter /u01/app/oracle/middleware for middleware home)
--(enter /u01/app/oracle/middleware/osb for product oracle home)

gtar -czvf osb_issue.tgz osb_issue

6. Send the zipped file to Oracle Support.

The post Collect WebLogic Server Diagnostic Data – RDA appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

Admin Server hangs during startup, the stdout doesn’t go beyond initialising security realm.

We took thread dump when Admin Server was stuck during startup and found the below thread connecting to the nodemanager.

[STANDBY] ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default (self-tuning)'” #24 daemon prio=5 os_prio=0 tid=0x00007f7720002000 nid=0x2be9 runnable [0x00007f7760db3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
– locked <0x00000000d57c80e8> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at weblogic.nodemanager.client.DefaultSSLClient.createSocket(DefaultSSLClient.java:32)
at weblogic.nodemanager.client.NMServerClient.connect(NMServerClient.java:452)
at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:424)
at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:430)
at weblogic.nodemanager.client.NMServerClient.initState(NMServerClient.java:411)
– locked <0x00000000d57c41b8> (a weblogic.nodemanager.client.DefaultSSLClient)
at weblogic.nodemanager.client.DefaultSSLClient.initState(DefaultSSLClient.java:14)
at weblogic.nodemanager.mbean.NodeManagerRuntime.initState(NodeManagerRuntime.java:425)
at weblogic.server.ServerLifeCycleRuntime.(ServerLifeCycleRuntime.java:134)
at weblogic.server.ServerLifeCycleService.createServerLifeCycleRuntime(ServerLifeCycleService.java:264)
at weblogic.server.ServerLifeCycleService.start(ServerLifeCycleService.java:150)

One investigating further we found one of the host was not responsive and we suspected that the Admin Server is not able to communicate with the node manager on that host. We looked for any such known issue with Weblogic Server and found the one below.

WLS 12.1.3.0.0 – Weblogic Admin Server Hangs While Starting Up (Doc ID 2241843.1)

For us the issue was resolved by fixing the host issue but we also plan to install the patch for this issue.

The post WebLogic Admin Server hangs during startup appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

You need to change the web-app version to 2.5  as previous version does not support JSP 2.1. Weblogic server enforces the jsp version mentioned in the web-app of the web.xml.

It should be like this.

<web-app
xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
version="2.5"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_5.xsd">

I have uploaded a sample application which can be dowloaded from here

The post trimDirectiveWhitespaces not working? appeared first on Middleware wonders!!.

Copyright 2013 - http://www. Weblogic-Wonders.com

javax.servlet.ServletException: java.lang.NoClassDefFoundError:

Application was failing to find an application class.
Using jarscan we searched the jars containing this class from the middleware home.
The jars were present in the domain lib.
We also checked the classpath in the startWeblogic.cmd and setDomainEnv.cmd to check if direct references were present in the server classpath.

The we logged in to Weblogic Console and analyzed the classloading conflict from the classloader tool.

You can find more details here.

http://docs.oracle.com/cd/E24329_01/web.1211/e24368/classloading.htm#BABHJGJJ

We kept the jar containing the class in the APP-INF lib of the EAR and added the following in the weblogic-application.xml

<prefer-application-packages>
        <package-name>com.demo.*</package-name>
</prefer-application-packages>

Using prefer-application-packages we can ensure that our application packages are loaded and not Weblogic Server’s.

After updating the application the issue was resolved.

The post Solving NoClassDefFoundError using the classloader analysis tool (CAT) appeared first on Middleware wonders!!.