AIX Grid Control 10.2.0.5 Communication and Monitoring Issue since 31-DEC-2010
- by jayatheertha.rao(at)oracle.com
Detailed symptoms for Oracle Management Server (OMS) 10.2.0.5 on AIX
Oracle Management Service 10.2.0.5 instances on AIX 5L remain active and functional, but the OMS instances fail to communicate with the Grid Control Management Agents.An SSLPeerUnverified exception will be reported in the file $ORACLE_HOME/sysman/log/emoms.trc when OMS attempts to connect with an Agent:Javax.net.ssl.SSLPeerUnverifiedException: peer not authenticatedat com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificateChain(DashoA12275)at oracle.sysman.emSDK.emd.comm.EMDClient.authenticateHTTPConnection(EMDClient.java:2002)at oracle.sysman.emSDK.emd.comm.EMDClient.getConnection(EMDClient.java:1877)at oracle.sysman.emSDK.emd.comm.EMDClient.getConnection(EMDClient.java:1810)at oracle.sysman.emSDK.emd.comm.EMDClient.verifyHttpConnection(EMDClient.java:2540)at oracle.sysman.emSDK.emd.comm.EMDClient.getResponseForRequest(EMDClient.java:2323)at oracle.sysman.emSDK.emd.comm.EMDClient.getUploadManagerStatus(EMDClient.java:4853)at oracle.sysman.eml.admin.rep.emdConfig.EmdConfigTargetsData.getEmdUploadData(EmdConfigTargetsData.java:1640)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)This error may be reported when:- Accessing the Agent home page in Grid Control- Setting preferred credentials for a target monitored by the Agent- Managing metrics for a target monitored by the Agent
The jobs scheduled to be run by Agents can become non-responsiveThe OMS log file $ORACLE_HOME/sysman/log/emoms.trc can show:2010-12-31 00:06:58,204 [JobWorker 430:Thread-34] DEBUG emSDK.comm getStreamResponse.4015 - oracle.sysman.emSDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticatedoracle.sysman.emSDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticatedat oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse_(EMDClient.java:4088)at oracle.sysman.emSDK.emd.comm.EMDClient.getStreamResponse(EMDClient.java:4009)at oracle.sysman.emSDK.emd.comm.EMDClient.remoteOperation(EMDClient.java:3404)at oracle.sysman.emdrep.jobs.CommandManager.requestRemoteCommand(CommandManager.java:765)at oracle.sysman.emdrep.jobs.commands.RemoteOp.executeCommand(RemoteOp.java:434)at oracle.sysman.emdrep.jobs.commands.RemoteOp.executeCommand(RemoteOp.java:491)at oracle.sysman.emdrep.jobs.BaseJobWorker.runStep(BaseJobWorker.java:614)at oracle.sysman.emdrep.jobs.BaseJobWorker.doOneOperation(BaseJobWorker.java:738)at oracle.sysman.emdrep.jobs.JobWorker.doOneOperation(JobWorker.java:306)at oracle.sysman.emdrep.jobs.JobWorker.run(JobWorker.java:288)at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:261)
Detailed symptoms for Grid Control Management Agent 10.2.0.5 on AIX
Beginning 31-DEC-2010 00:00:00, 10.2.0.5 Management Agents running on the AIX 5L operating system will fail to monitor Oracle Application Server targets. As a result, the Availability Status for the Oracle Application Server targets will be in the "Metric Error" state.
NOTE: The 10.2.0.5.0 Agents would experience these errors regardless of the version/platform of the OMS.The following metric error is seen in the console for the Oracle Application Server targets monitored by a Grid Control Management Agent 10.2.0.5 installed on AIX and experiencing a Root Certificate Authority issue:Message oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated The Grid Control Management Agent log file $ORACLE_HOME/sysman/log/emagentfetchlet.log (or $ORACLE_HOME/hostname/sysman/log/emagentfetchlet.log for a clustered Agent) includes the following errors:2010-12-31 00:01:03,626 [nmefmgr_getJNIFetchlet] ERROR ias.ResponseMetric getResponseMetric.154 - Unable tocompute application server statusoracle.sysman.emSDK.emd.fetchlet.FetchletException: oracle.sysman.emSDK.emd.comm.CommException: java.io.IOException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticatedat oracle.sysman.ias.ias.ResponseMetric.getResponseMetric(ResponseMetric.java:108)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:618)at oracle.sysman.emd.fetchlets.JavaWrapperFetchlet.getMetric(JavaWrapperFetchlet.java:217)at oracle.sysman.emd.fetchlets.FetchletWrapper.getMetric(FetchletWrapper.java:382)
Beginning 31-DEC-2010, 10.2.0.5 Management Agents on the AIX 5L platform will fail to secure or re-secure with Oracle Management Service (OMS). This failure will cause installation of 10.2.0.5 Agents on the AIX 5L platform to fail.NOTE: The 10.2.0.5.0 Agents would experience these errors regardless of the version/platform of the OMS.The "emctl secure agent" command will fail with the following error, which will be written to the $ORACLE_HOME/sysman/log/secure.log file (or $ORACLE_HOME/hostname/sysman/log/secure.log for a clustered Agent) :2011-01-03 21:06:11,941 [main] ERROR agent.SecureAgentCmd main.207 - Failedto secure the Agent:javax.net.ssl.SSLPeerUnverifiedException: peer not authenticatedatcom.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificateChain(DashoA6275)atoracle.sysman.emctl.secure.agent.SecureAgentCmd.checkUpload(SecureAgentCmd.java:478)atoracle.sysman.emctl.secure.agent.SecureAgentCmd.secureAgent(SecureAgentCmd.java:249)atoracle.sysman.emctl.secure.agent.SecureAgentCmd.main(SecureAgentCmd.java:200)
For solution, refer to AIX Grid Control 10.2.0.5 SSL Communication and Monitoring Issue since 31-DEC-2010 (Doc ID 1275070.1)