What tools are available for diagnosing SOA Suite issues?
There are a variety of tools available to help you and Support diagnose SOA Suite issues in 11g but it can be confusing as to which tool is appropriate for a particular situation and what their
relationships are. This blog post will introduce
the various tools and attempt to clarify what each is for and how they are related. Let's first list
the tools we'll be addressing:
RDA: Remote Diagnostic Agent
DFW: Diagnostic Framework
Selective Tracing
DMS: Dynamic Monitoring Service
ODL: Oracle Diagnostic Logging
ADR: Automatic Diagnostics Repository
ADRCI: Automatic Diagnostics Repository Command Interpreter
WLDF: WebLogic Diagnostic Framework
This overview is not mean to be a comprehensive guide on using all of these tools, however, extensive reference materials are included that will provide many more details on their execution. Another
point to note is that all of these tools are applicable for Fusion Middleware as a whole but specific products may or may not have implemented features to leverage them.
A couple of
the tools have a WebLogic Scripting Tool or 'WLST' interface. WLST is a command interface for executing pre-built functions and custom scripts against a domain. A detailed
WLST tutorial is beyond
the scope of this post but you can find general information here.
There are more specific resources in
the below sections.
In this post when we refer to 'Enterprise Manager' or 'EM' we are referring to Enterprise Manager Fusion Middleware Control.
RDA (Remote Diagnostic Agent)
RDA is a standalone tool that is used to collect both static configuration and dynamic runtime information from
the SOA environment. RDA is generally
run manually from
the command line against a domain or single server. When opening a new Service Request, including an RDA collection can dramatically decrease
the back and forth required to
collect logs and configuration information for Support.
After installing RDA you configure it to use
the SOA Suite module as decribed in
the referenced resources.
The SOA module includes
the Oracle WebLogic Server (WLS) module by default in order to include all of
the relevant
information for
the environment. In addition to this basic configuration there is also an advanced mode where you can set
the number of thread dumps for
the collections, log files, Incidents, etc.
When would you use it?
When creating a Service Request or otherwise working with Oracle resources on an issue, capturing environment snapshots to baseline your configuration or to diagnose an issue on your own.
How is it related to
the other tools?
RDA is related to DFW in that it collects
the last 10 Incidents from
the server by default. In a similar manner, RDA is related to ODL through its collection of
the diagnostic logs and these may
contain information from Selective Tracing sessions.
Examples of what it currently collects: (for details please see
the links in
the Resources section)
Diagnostic Logs (ODL)
Diagnostic Framework Incidents (DFW)
SOA MDS Deployment Descriptors
SOA Repository Summary Statistics
Thread Dumps
Complete Domain Configuration
RDA Resources:
Webcast Recording: Using RDA with Oracle SOA Suite 11g
Blog Post: Diagnose SOA Suite 11g Issues Using RDA
Download RDA
How to Collect Analysis Information Using RDA for Oracle SOA Suite 11g
Products [ID 1350313.1]
How to Collect Analysis Information Using RDA for Oracle SOA Suite and
BPEL Process Manager 11g [ID 1352181.1]
Getting Started With Remote Diagnostic Agent: Case Study - Oracle
WebLogic Server (Video) [ID 1262157.1]
top
DFW (Diagnostic Framework)
DFW provides
the ability to collect specific information for a particular problem when that problem occurs. DFW is included with your SOA Suite installation and deployed to
the domain.
Let's define
the components of DFW.
Diagnostic Dumps: Specific diagnostic
collections that are defined at either
the 'system' or product level. Examples would be diagnostic logs or thread dumps.
Incident: A collection of Diagnostic Dumps associated with a particular problem
Log Conditions: An Oracle Diagnostic Logging event that DFW is configured to listen for. If
the event is identified then an Incident will be created.
WLDF Watch:
The WebLogic Diagnostic Framework or 'WLDF' is not a component of DFW, however, it can be a source of DFW Incident creation through
the use of a 'Watch'.
WLDF Notification: A Notification is a component of WLDF and is
the link between
the Watch and DFW. You can configure multiple Notification types in WLDF and associate them
with your Watches. 'FMWDFW-notification' is available to you out of
the box to allow for DFW notification of Watch execution.
Rule: Defines a WLDF Watch or Log Condition for which we want to associate a set of Diagnostic Dumps. When triggered
the specified dumps will be collected and added to
the
Incident
Rule Action: Defines
the specific Diagnostic Dumps to collect for a particular rule
ADR: Automatic Diagnostics Repository; Defined for every server in a domain. This is where Incidents are stored
Now let's walk through a simple flow:
Oracle Web Services error message OWS-04086 (SOAP Fault) is generated on managed server 1
DFW Log Condition for OWS-04086 evaluates to TRUE
DFW creates a new Incident in
the ADR for managed server 1
DFW executes
the specified Diagnostic Dumps and adds
the output to
the Incident
In this case we'll grab
the diagnostic log and thread dump. We might also want to collect
the WSDL binding information and SOA audit trail
When would you use it?
When you want to automatically collect Diagnostic Dumps at a particular time using a trigger or when you want to manually collect
the information. In either case it can be readily uploaded to
Oracle Support through
the Service Request.
How is it related to
the other tools?
DFW generates Incidents which are
collections of Diagnostic Dumps. One of
the system level Diagonstic Dumps collects
the current server diagnostic log which is generated by ODL and can contain
information from Selective Tracing sessions. Incidents are included in RDA
collections by default and ADRCI is a tool that is used to package an Incident for upload to Oracle Support. In
addition, both ODL and DMS can be used to trigger Incident creation through DFW.
The conditions and rules for generating Incidents can become quite complicated and
the below resources go into more detail. A simpler approach to leveraging at least
the Diagnostic Dumps is through
WLST (WebLogic Scripting Tool) where there are commands to do
the following:
Create an Incident
Execute a single Diagnostic Dump
Describe a Diagnostic Dump
List
the available Diagnostic Dumps
The WLST option offers greater control in what is generated and when. It can be a great help when collecting information for Support. There are overlaps with RDA, however, DFW is geared
towards collecting specific runtime information when an issue occurs while existing Incidents are collected by RDA.
There are 3 WLDF Watches configured by default in a SOA Suite 11g domain: Stuck Threads, Unchecked Exception and Deadlock.
These Watches are enabled by default and will generate Incidents in ADR. They are configured to reset automatically after 30 seconds so they have
the potential to create multiple Incidents if these
conditions are consistent.
The Incidents generated by these Watches will only contain System level Diagnostic Dumps. These same System level Diagnostic Dumps will be included in any application
scoped Incident as well.
Starting in 11.1.1.6, SOA Suite is including its own set of application scoped Diagnostic Dumps that can be executed from WLST or through a WLDF Watch or Log Condition. These Diagnostic Dumps can be
added to an Incident such as in
the earlier example using
the error code OWS-04086.
soa.config: MDS configuration files and deployed-composites.xml
soa.composite: All artifacts related to
the deployed composite
soa.wsdl: Summary of endpoints configured for
the composite
soa.edn: EDN configuration summary if applicable
soa.db: Summary DB information for
the SOA repository
soa.env: Coherence cluster configuration summary
soa.composite.trail: Partial audit trail information for
the running composite
The current release of RDA has
the option to collect
the soa.wsdl and soa.composite Diagnostic Dumps. More Diagnostic Dumps for SOA Suite products are planned for future
releases along with enhancements to DFW itself.
DFW Resources:
Webcast Recording: SOA Diagnostics Sessions: Diagnostic Framework
Diagnostic Framework Documentation
DFW WLST Command Reference
Documentation for SOA Diagnostic Dumps in 11.1.1.6
top
Selective Tracing
Selective Tracing is a facility available starting in version 11.1.1.4 that allows you to increase
the logging level for specific loggers and for a specific context. What this means is that you have
greater capability to collect needed diagnostic log information in a production environment with reduced overhead. For example, a Selective Tracing session can be executed that only increases
the log level for one composite, only one logger, limited to one server in
the cluster and for a preset period of time. In an environment where dozens of composites are deployed this can dramatically
reduce
the volume and overhead of
the logging without sacrificing relevance.
Selective Tracing can be administered either from Enterprise Manager or through WLST. WLST provides a bit more flexibility in terms of exactly where
the tracing is run.
When would you use it?
When there is an issue in production or another environment that lends itself to filtering by an available context criteria and increasing
the log level globally results in too much overhead or
irrelevant information.
The information is written to
the server diagnostic log and is exportable from Enterprise Manager
How is it related to
the other tools?
Selective Tracing output is written to
the server diagnostic log. This log can be collected by a system level Diagnostic Dump using DFW or through a default RDA collection. Selective Tracing also
heavily leverages ODL fields to determine what to trace and to tag information that is part of a particular tracing session.
Available Context Criteria:
Application Name
Client Address
Client Host
Composite Name
User Name
Web Service Name
Web Service Port
Selective Tracing Resources:
Webcast Recording: SOA Diagnostics Session: Using Selective Tracing to Diagnose SOA Suite Issues
How to Use Selective Tracing for SOA [ID 1367174.1]
Selective Tracing WLST Reference
top
DMS (Dynamic Monitoring Service)
DMS exposes runtime information for monitoring. This information can be monitored in two ways:
Through
the DMS servlet
As exposed MBeans
The servlet is deployed by default and can be accessed through http://<host>:<port>/dms/Spy (use administrative credentials to access).
The landing page of
the servlet shows identical columns of what
are known as Noun Types. If you select a Noun Type you will see a table in
the right frame that shows
the attributes (Sensors) for
the Noun Type and
the available instances. SOA Suite has several exposed Noun
Types that are available for viewing through
the Spy servlet. Screenshots of
the Spy servlet are available in
the Knowledge Base article
How to Monitor Runtime SOA Performance With
the Dynamic Monitoring Service (DMS).
Every Noun instance in
the runtime is exposed as an MBean instance. As such they are generally available through an MBean browser and available for monitoring through WLDF. You can configure a
WLDF Watch to monitor a particular attribute and fire a notification when
the threshold is exceeded. A WLDF Watch can use
the out of
the box DFW notification type to notify DFW to create an Incident.
When would you use it?
When you want to monitor a metric or set of metrics either manually or through an automated system. When you want to trigger a WLDF Watch based on a metric exposed through DMS.
How is it related to
the other tools?
DMS metrics can be monitored with WLDF Watches which can in turn notify DFW to create an Incident.
DMS Resources:
How to Monitor Runtime SOA Performance With
the Dynamic Monitoring Service (DMS) [ID 1368291.1]
How to Reset a SOA 11g DMS Metric
DMS Documentation
top
ODL (Oracle Diagnostic Logging)
ODL is
the primary facility for most Fusion Middleware applications to log what they are doing. Whenever you change a logging level through Enterprise Manager it is ultimately exposed through ODL and
written to
the server diagnostic log. A notable exception to this is WebLogic Server which uses its own log format / file.
ODL logs entries in a consistent, structured way using predefined fields and name/value pairs. Here's an example of a SOA Suite entry:
[2012-04-25T12:49:28.083-06:00] [AdminServer] [ERROR] [] [oracle.soa.bpel.engine] [tid: [ACTIVE].ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: ] [ecid: 0963fdde7e77631c:-31a6431d:136eaa46cda:-8000-00000000000000b4,0] [errid: 41] [WEBSERVICE_PORT.name: BPELProcess2_pt] [APP: soa-infra] [composite_name: TestProject2] [J2EE_MODULE.name: fabric] [WEBSERVICE.name: bpelprocess1_client_ep] [J2EE_APP.name: soa-infra] Error occured while handling a post operation[[
When would you use it?
You'll use ODL almost every time you want to identify and diagnose a problem in
the environment.
The entries are written to
the server diagnostic log.
How is it related to
the other tools?
The server diagnostic logs are collected by DFW and RDA. Selective Tracing writes its information to
the diagnostic log as well.
Additionally, DFW log conditions are triggered by ODL log events.
ODL Resources:
ODL Documentation
top
ADR (Automatic Diagnostics Repository)
ADR is not a tool in and of itself but is where DFW stores
the Incidents it creates. Every server in
the domain has an ADR location which can be found under <SERVER_HOME>/adr. This is referred to
the as
the ADR 'Base' location.
ADR also has what are known as 'Home' locations. Example:
You have a domain called 'myDomain' and an associated managed server called 'myServer'. Your admin server is called 'AdminServer'.
Your domain home directory is called 'myDomain' and it contains a 'servers' directory.
The 'servers' directory contains a directory for
the managed server called 'myServer' and here is where you'll find
the 'adr' directory which is
the ADR 'Base' location for myServer.
To get to
the ADR 'Home' locations we drill through a few levels: diag/ofm/myDomain/
In an 11.1.1.6 SOA Suite domain you will see 2 directories here, 'myServer' and 'soa-infra'. These are
the ADR 'Home' locations.
'myServer' is
the 'system' ADR home and contains system level Incidents.
'soa-infra' is
the name that SOA Suite used to register with DFW and this ADR home contains SOA Suite related Incidents
Each ADR home location contains a series of directories, one of which is called 'incident'. This is where your Incidents are stored.
When would you use it?
It's a good idea to check on these locations from time to time to see whether a lot of Incidents are being generated. They can be cleaned out by deleting
the Incident directories or through
the ADRCI
tool. If you know that an Incident is of particular interest for an issue you're working with Oracle you can simply zip it up and provide it.
How does it relate to
the other tools?
ADR is obviously very important for DFW since it's where
the Incidents are stored. Incidents contain Diagnostic Dumps that may relate to diagnostic logs (ODL) and DMS metrics.
The most recent 10 Incident directories are collected by RDA by default and ADRCI relies on
the ADR locations to help manage
the contents.
top
ADRCI (Automatic Diagnostics Repository Command Interpreter)
ADRCI is a command line tool for packaging and managing Incidents.
When would you use it?
When purging Incidents from an ADR Home location or when you want to package an Incident along with an offline RDA collection for upload to Oracle Support.
How does it relate to
the other tools?
ADRCI contains a tool called
the Incident Packaging System or IPS. This is used to package an Incident for upload to Oracle Support through a Service Request. Starting in 11.1.1.6 IPS will attempt
to collect an offline RDA collection and include it with
the Incident package. This will only work if Perl is available on
the path, otherwise it will give a warning and package only
the Incident
files.
ADRCI Resources:
How to Use
the Incident Packaging System (IPS) in SOA 11g [ID 1381259.1]
ADRCI Documentation
top
WLDF (WebLogic Diagnostic Framework)
WLDF is functionality available in WebLogic Server since version 9. Starting with FMw 11g a link has been added between WLDF and
the pre-existing DFW,
the WLDF Watch Notification. Let's take a closer look
at
the flow:
There is a need to monitor
the performance of your SOA Suite message processing
A WLDF Watch is created in
the WLS console that will trigger if
the average message processing time exceeds 2 seconds. This metric is monitored through a DMS MBean instance.
The out of
the box DFW Notification (the Notification is called FMWDFW-notification) is added to
the Watch. Under
the covers this notification is of type JMX.
The Watch is triggered when
the threshold is exceeded and fires
the Notification.
DFW has a listener that picks up
the Notification and evaluates it according to its rules, etc
When it comes to automatic Incident creation, WLDF is a key component with capabilities that will grow over time.
When would you use it?
When you want to monitor
the WLS server log or an MBean metric for some condition and fire a notification when
the Watch is triggered.
How does it relate to
the other tools?
WLDF is used to automatically trigger Incident creation through DFW using
the DFW Notification.
WLDF Resources:
How to Monitor Runtime SOA Performance With
the Dynamic Monitoring Service (DMS) [ID 1368291.1]
How To Script
the Creation of a SOA WLDF Watch in 11g [ID 1377986.1]
WLDF Documentation
top
