Using BPEL Performance Statistics to Diagnose Performance Bottlenecks
- by fip
Tuning performance of Oracle SOA 11G applications could be challenging.
Because SOA is a platform for you to build composite applications that
connect many applications and "services", when the overall performance
is slow, the bottlenecks could be anywhere in the system: the
applications/services that SOA connects to, the infrastructure database,
or the SOA server itself.How to quickly identify the bottleneck
becomes crucial in tuning the overall performance.
Fortunately, the BPEL engine in Oracle SOA 11G (and 10G, for that
matter) collects BPEL Engine Performance Statistics, which show the
latencies of low level BPEL engine activities. The BPEL engine
performance statistics can make it a bit easier for you to identify the
performance bottleneck.
Although the BPEL engine performance statistics are always available,
the access to and interpretation of them are somewhat obscure in the
early and current (PS5) 11G versions.
This blog attempts to offer instructions that help you to enable,
retrieve and interpret the performance statistics, before the future
versions provides a more pleasant user experience.
Overview of BPEL Engine Performance Statistics
SOA BPEL has a feature of collecting some performance statistics and store them in memory.
One MBean attribute, StatLastN, configures the size of the memory buffer
to
store the statistics. This memory buffer is a "moving window", in a way
that old statistics will be flushed out by the new if the amount of
data exceeds the buffer size. Since the buffer size is limited by
StatLastN, impacts of statistics collection on performance is minimal.
By default StatLastN=-1, which means no
collection of performance data.
Once the statistics are collected in the memory buffer, they can be retrieved via another MBean
oracle.as.soainfra.bpel:Location=[Server
Name],name=BPELEngine,type=BPELEngine.>
My friend in Oracle SOA development wrote this simple 'bpelstat' web app that looks up and retrieves the performance
data from the MBean and displays it in a human readable form. It does not have beautiful UI but it is fairly useful.
Although in Oracle SOA 11.1.1.5 onwards the same statistics can be
viewed via a more elegant UI under "request break down" at EM -> SOA
Infrastructure
-> Service Engines -> BPEL -> Statistics, some unsophisticated
minds like mine may still prefer the simplicity of the 'bpelstat' JSP.
One thing that simple JSP does do well is that you can save the page and
send it to someone to further analyze
Follows are the instructions of how to install and invoke the BPEL
statistic JSP. My friend in SOA Development will soon blog about
interpreting the statistics. Stay tuned.
Step1: Enable BPEL Engine Statistics for Each SOA Servers via Enterprise Manager
First st you need to set the StatLastN to some number as a way to enable the collection of BPEL Engine Performance Statistics
EM Console -> soa-infra(Server Name) -> SOA Infrastructure -> SOA Administration -> BPEL Properties
Click on "More BPEL Configuration Properties"
Click on attribute "StatLastN", set its value to some integer number. Typically you want to set it 1000 or more.
Step 2: Download and Deploy bpelstat.war File to Admin Server, Note:
the WAR file contains a JSP that does NOT have any security
restriction. You do NOT want to keep in your production server for a
long time as it is a security hazard. Deactivate the war once you are
done.
Download the bpelstat.war to your local PC
At WebLogic Console, Go to Deployments -> Install
Click on the "upload your file(s)"
Click the "Browse" button to upload the deployment to Admin Server
Accept the uploaded file as the path, click next
Check the default option "Install this deployment as an application"
Check "AdminServer" as the target server
Finish the rest of the deployment with default settings
Console -> Deployments
Check the box next to "bpelstat" application
Click on the "Start" button. It will change the state of the app from "prepared" to "active"
Step 3: Invoke the BPEL Statistic Tool
The
BPELStat tool merely call the MBean of BPEL server and collects and
display the in-memory performance statics. You usually want to do that
after some peak loads.
Go to http://<admin-server-host>:<admin-server-port>/bpelstat
Enter the correct admin hostname, port, username and password
Enter the SOA Server Name from which you want to collect the performance statistics. For example, SOA_MS1, etc.
Click Submit
Keep doing the same for all SOA servers.
Step 3: Interpret the BPEL Engine Statistics You will see a few categories of BPEL Statistics from the JSP Page. First
it starts with the overall latency of BPEL processes, grouped by
synchronous and asynchronous processes. Then it provides the further
break down of the measurements through the life time of a BPEL request,
which is called the "request break down". 1. Overall latency of BPEL processes
The top of the page shows that the elapse time of executing the
synchronous process TestSyncBPELProcess from the composite TestComposite
averages at about 1543.21ms, while the elapse time of executing the
asynchronous process TestAsyncBPELProcess
from the composite TestComposite2 averages at about 1765.43ms. The
maximum and minimum latency were also shown.
Synchronous process statistics
<statistics>
<stats
key="default/TestComposite!2.0.2-ScopedJMSOSB*soa_bfba2527-a9ba-41a7-95c5-87e49c32f4ff/TestSyncBPELProcess"
min="1234" max="4567" average="1543.21" count="1000">
</stats>
</statistics>
Asynchronous process statistics
<statistics>
<stats
key="default/TestComposite2!2.0.2-ScopedJMSOSB*soa_bfba2527-a9ba-41a7-95c5-87e49c32f4ff/TestAsyncBPELProcess"
min="2234" max="3234" average="1765.43" count="1000">
</stats>
</statistics>
2. Request break down
Under the overall latency categorized by synchronous and asynchronous
processes is the "Request breakdown". Organized by statistic keys, the
Request breakdown gives finer grain performance statistics
through the life time of the BPEL requests.It uses indention to show
the hierarchy of the statistics.
Request breakdown
<statistics>
<stats key="eng-composite-request" min="0" max="0" average="0.0" count="0">
<stats
key="eng-single-request" min="22" max="606" average="258.43"
count="277">
<stats
key="populate-context" min="0" max="0" average="0.0" count="248">
Please
note that in SOA 11.1.1.6, the statistics under Request breakdown is
aggregated together cross all the BPEL processes based on statistic
keys. It does not differentiate between BPEL processes. If two BPEL
processes happen to have the statistic that share same statistic key,
the statistics from two BPEL processes will be aggregated together. Keep
this in mind when we go through more details below. 2.1 BPEL process activity latencies A
very useful measurement in the Request Breakdown is the performance
statistics of the BPEL activities you put in your BPEL processes:
Assign, Invoke, Receive, etc. The names of the measurement in the JSP
page directly come from the names to assign to each BPEL activity. These
measurements are under the statistic key "actual-perform" Example 1: Follows
is the measurement for BPEL activity
"AssignInvokeCreditProvider_Input", which looks like the Assign activity
in a BPEL process that assign an input variable before passing it to
the invocation:
<stats
key="AssignInvokeCreditProvider_Input" min="1" max="8" average="1.9"
count="153">
<stats
key="sensor-send-activity-data" min="0" max="1" average="0.0"
count="306">
</stats>
<stats
key="sensor-send-variable-data" min="0" max="0" average="0.0"
count="153">
</stats>
<stats
key="monitor-send-activity-data" min="0" max="0" average="0.0"
count="306">
</stats>
</stats>
Note: because as previously mentioned that the statistics cross all BPEL
processes are aggregated together based on statistic keys, if two BPEL
processes happen to name their Invoke activity the same name, they will
show up at one measurement (i.e. statistic key). Example 2: Follows
is the measurement of BPEL activity called "InvokeCreditProvider". You
can not only see that by average it takes 3.31ms to finish this call
(pretty fast) but also you can see from the further break down that most
of this 3.31 ms was spent on the "invoke-service".
<stats
key="InvokeCreditProvider" min="1" max="13" average="3.31"
count="153">
<stats
key="initiate-correlation-set-again" min="0" max="0" average="0.0"
count="153">
</stats>
<stats
key="invoke-service" min="1" max="13" average="3.08" count="153">
<stats
key="prep-call" min="0" max="1" average="0.04" count="153">
</stats>
</stats>
<stats
key="initiate-correlation-set" min="0" max="0" average="0.0"
count="153">
</stats>
<stats
key="sensor-send-activity-data" min="0" max="0" average="0.0"
count="306">
</stats>
<stats
key="sensor-send-variable-data" min="0" max="0" average="0.0"
count="153">
</stats>
<stats
key="monitor-send-activity-data" min="0" max="0" average="0.0"
count="306">
</stats>
<stats
key="update-audit-trail" min="0" max="2" average="0.03" count="153">
</stats>
</stats>
2.2 BPEL engine activity latency Another
type of measurements under Request breakdown are the latencies of
underlying system level engine activities. These activities are not
directly tied to a particular BPEL process or process activity, but they
are critical factors in the overall engine performance. These
activities include the latency of saving asynchronous requests to
database, and latency of process dehydration. My
friend Malkit Bhasin is working on providing more information on
interpreting the statistics on engine activities on his blog (https://blogs.oracle.com/malkit/). I will update this blog once the information becomes available.
Update on 2012-10-02: My friend Malkit Bhasin has published the detail
interpretation of the BPEL service engine statistics at his blog http://malkit.blogspot.com/2012/09/oracle-bpel-engine-soa-suite.html.
