Queued Processing to remove Concurrency issues in Loadtest Scripts

Some scripts act on information returned by the server, e.g. act on first item in the returned list of pending tasks/actions. This may lead to concurrency issues if the virtual users simulated in a load test scenario are not synchronized in some way.

As the load test cases should be carried out in a comparable and straight forward manner simply cancel a transaction in case a collision occurs is clearly not an option. In case you increase the number of virtual users this approach would lead to a high number of requests for the early steps in your transaction (e.g. login, retrieve list of action points, assign an action point to the virtual user) but later steps would be rarely visited successfully or at all, depending on the application logic.



A way to tackle this problem is to enqueue the virtual users in a Shared Data Service queue. Only the first virtual user in this queue will be allowed to carry out the critical steps (retrieve list of action points, assign an action point to the virtual user) in your transaction at any one time.

Once a virtual user has passed the critical path it will dequeue himself from the head of the queue and continue with his actions. This does theoretically allow virtual users to run in parallel all steps of the transaction which are not part of the critical path.

In practice it has been seen this is rarely the case, though it does not allow adding more than N users to perform a transaction without causing delays due to virtual users waiting in the queue. N being the time of the total transaction divided by the sum of the time of all critical steps in this transaction.

While this problem can be circumvented by allowing multiple queues to act on individual segments of the list of actions, e.g. per country filter, ends with 0..9 filter, etc.

This would require additional handling of these additional queues of slots for the virtual users at the head of the queue in order to maintain the mutually exclusive access to the first element in the list returned by the server at any one time of the load test. Such an improved handling of multiple queues and/or multiple slots is above the subject of this paper.

Shared Data Services Pre-Requisites

Start WebLogic Server to host Shared Data Services

You will have to make sure that your WebLogic server is installed and started. Shared Data Services may not work if you installed only the minimal installation package for OpenScript. If however you installed the default package including OLT and OTM, you may follow the instructions below to start and verify WebLogic installation.

To start the WebLogic Server deployed underneath of Oracle Load Testing and/or Oracle Test Manager you can go to your Start menu, Oracle Application Testing Suite and select the Restart Oracle Application Testing Suite Application Service entry from the Tools submenu.



To verify the service has been started you can run the Microsoft Management Console for Services by Selecting Run from the Start Menu and entering services.msc. Look for the entry that reads Oracle Application Testing Suite Application Service, once it has changed it status from Starting to Started you can proceed to verify the login. Please note that this may take several minutes, I would say up to 10 minutes depending on the strength of your CPU horse-power.



Verify WebLogic Server user credentials

You will have to make sure that your WebLogic Server is installed and started. Next open the Oracle WebLogic Server Adminstration Console on http://localhost:8088/console.

It may take a while until the application is deployed and started. It may display the following until the Administration Console has been deployed on the fly.



Afterwards you can login using the username oats and the password that you selected during install time for your Application Testing Suite administrative purposes.



This will bring up the Home page of you WebLogic Server. You have actually verified that you are able to login with these credentials already. However if you want to check the details, navigate to Security Realms, myrealm, Users and Groups tab.

Here you could add users to your WebLogic Server which could be used in the later steps. Details on the Groups required for such a custom user to work are exceeding this quick overview and have to be selected with the WebLogic Server Adminstration Guide in mind.



Shared Data Services pre-requisites for Load testing

OpenScript Preferences have to be set to enable Encryption and provide a default Shared Data Service Connection for Playback.

These are pre-requisites you want to use for load testing with Shared Data Services.

Please note that the usage of the Connection Parameters (individual directive in the script) for Shared Data Services did not playback reliably in the current version 9.20.0370 of Oracle Load Testing (OLT) and encryption of credentials still seemed to be mandatory as well.

General Encryption settings

Select OpenScript Preferences from the View menu and navigate to the General, Encryption entry in the tree on the left. Select the Encrypt script data option from the list and enter the same password that you used for securing your WebLogic Server Administration Console.



Enable global shared data access credentials

Select OpenScript Preferences from the View menu and navigate to the Playback, Shared Data entry in the tree on the left. Enable the global shared data access credentials and enter the Address, User name and Password determined for your WebLogic Server to host Shared Data Services.



Please note, that you may want to replace the localhost in Address with the hosts realname in case you plan to run load tests with Loadtest Agents running on remote systems.

Queued Processing of Transactions

Enable Shared Data Services Module in Script Properties

The Shared Data Services Module has to be enabled for each Script that wants to employ the Shared Data Service Queue functionality in OpenScript. It can be enabled under the Script menu selecting Script Properties.

On the Script Properties Dialog select the Modules section and check Shared Data to enable Shared Data Service Module for your script.

Checking the Shared Data Services option will effectively add a line to your script code that adds the sharedData ScriptService to your script class of IteratingVUserScript.

@ScriptService oracle.oats.scripting.modules.sharedData.api.SharedDataService sharedData;

Record your script

Record your script as usual and then add the following things for Queue handling in the Initialize code block, before the first step and after the last step of your critical path and in the Finalize code block.

The java code to be added at individual locations is explained in the following sections in full detail.



Create a Shared Data Queue in Initialize

To create a Shared Data Queue go to the Java view of your script and enter the following statements to the initialize() code block.

info("Create queueA with life time of 120 minutes");
sharedData.createQueue("queueA", 120);

This will create an instantiation of the Shared Data Queue object named queueA which is maintained for upto 120 minutes.

If you want to use the code for multiple scripts, make sure to use a different queue name for each one here and in the subsequent steps. You may even consider to use a dynamic queueName based on filters of your result list being concurrently accessed.

Prepare a unique id for each Iteration

In order to keep track of individual virtual users in our queue we need to create a unique identifier from the virtual user id and the used username right after retrieving the next record from our databank file.

getDatabank("Usernames").getNextDatabankRecord();
getVariables().set("usernameValue1","VU_{{@vuid}}_{{@iterationnum}}_{{db.Usernames.Username}}_{{@timestamp}}_{{@random(10000)}}");
String usernameValue = getVariables().get("usernameValue1");
info("Now running virtual user " + usernameValue);

As you can see from the above code block, we have set the OpenScript variable usernameValue1 to VU_{{@vuid}}_{{@iterationnum}}_{{db.Usernames.Username}}_{{@timestamp}}_{{@random(10000)}} which is a concatenation of the virtual user id and the iterationnumber for general uniqueness; as well as the username from our databank, the timestamp and a random number for making it further unique and ease spotting of errors.

Not all of these fields are actually required to make it really unique, but adding the queue name may also be considered to help troubleshoot multiple queues.

The value is then retrieved with the getVariables.get() method call and assigned to the usernameValue String used throughout the script.

Please note that moving the getDatabank("Usernames").getNextDatabankRecord(); call to the initialize block was later considered to remove concurrency of multiple virtual users running with the same userid and therefor accessing the same "My Inbox" in step 6. This will effectively give each virtual user a userid from the databank file. Make sure you have enough userids to remove this second hurdle.

Enqueue and attend Queue before Critical Path

To maintain the right order of virtual users being allowed into the critical path of the transaction the following pseudo step has to be added in front of the first critical step. In the case of this example this is right in front of the step where we retrieve the list of actions from which we select the first to be assigned to us.

beginStep("[0] Waiting in the Queue", 0);
{
info("Enqueued virtual user " + usernameValue + " at the end of queueA");
sharedData.offerLast("queueA", usernameValue);
info("Wait until the user is the first in queueA");
String queueValue1 = null;
do {
// we wait for at least 0.7 seconds before we check the head of the
// queue. This is the time it takes one user to move through the
// critical path, i.e. pass steps [5] Enter country and [6] Assign
// to me
Thread.sleep(700);
queueValue1 = (String) sharedData.peekFirst("queueA");
info("The first user in queueA is currently: '" + queueValue1 + "' " + queueValue1.getClass() + " length " + queueValue1.length() );
info("The current user is '"+ usernameValue + "' " + usernameValue.getClass() + " length " + usernameValue.length() + ": indexOf " + usernameValue.indexOf(queueValue1) + " equals " + usernameValue.equals(queueValue1) );
} while ( queueValue1.indexOf(usernameValue) < 0 );
info("Now the user is the first in queueA");
}
endStep();

This will enqueue the username to the tail of our Queue. It will will wait for at least 700 milliseconds, the time it takes for one user to exit the critical path and then compare the head of our queue with it's username. This last step will be repeated while the two are not equal (indexOf less than zero). If they are equal the indexOf will yield a value of zero or larger and we will perform the critical steps.

Dequeue after Critical Path

After the virtual user has left the critical path and complete its last step the following code block needs to dequeue the virtual user. In the case of our example this is right after the action has been actually assigned to the virtual user. This will allow the next virtual user to retrieve the list of actions still available and in turn let him make his selection/assignment.

info("Get and remove the current user from the head of queueA");
String pollValue1 = (String) sharedData.pollFirst("queueA");

The current user is removed from the head of the queue. The next one will now be able to match his username against the head of the queue.

Clear and Destroy Queue for Finish

When the script has completed, it should clear and destroy the queue. This code block can be put in the finish block of your script and/or in a separate script in order to clear and remove the queue in case you have spotted an error or want to reset the queue for some reason.

info("Clear queueA");
sharedData.clearQueue("queueA");
info("Destroy queueA");
sharedData.destroyQueue("queueA");

The users waiting in queueA are cleared and the queue is destroyed. If you have scripts still executing they will be caught in a loop.

I found it better to maintain a separate Reset Queue script which contained only the following code in the initialize() block. I use to call this script to make sure the queue is cleared in between multiple Loadtest runs. This script could also even be added as the first in a larger scenario, which would execute it only once at very start of the Loadtest and make sure the queues do not contain any stale entries.

info("Create queueA with life time of 120 minutes");
sharedData.createQueue("queueA", 120);
info("Clear queueA");
sharedData.clearQueue("queueA");

This will create a Shared Data Queue instance of queueA and clear all entries from this queue.

Monitoring Queue

While creating the scripts it was useful to monitor the contents, i.e. the current first user in the Queue. The following code block will make sure the Shared Data Queue is accessible in the initialize() block.

info("Create queueA with life time of 120 minutes");
sharedData.createQueue("queueA", 120);

In the run() block the following code will continuously monitor the first element of the Queue and write an informational message with the current username Value to the Result window.

info("Monitor the first users in queueA");
String queueValue1 = null;
do {
queueValue1 = (String) sharedData.peekFirst("queueA");
if (queueValue1 != null)
info("The first user in queueA is currently: '" + queueValue1 + "' " + queueValue1.getClass() + " length " + queueValue1.length() );
} while ( true );

This script can be run from OpenScript parallel to a loadtest performed by the Oracle Load Test.

However it is not recommend to run this in a production loadtest as the performance impact is unknown. Accessing the Queue's head with the peekFirst() method has been reported with about 2 seconds response time by both OpenScript and OTL. It is advised to log a Service Request to see if this could be lowered in future releases of Application Testing Suite, as the pollFirst() and even offerLast() writing to the tail of the Queue usually returned after an average 0.1 seconds.

Debugging Queue

While debugging the scripts the following was useful to remove single entries from its head, i.e. the current first user in the Queue. The following code block will make sure the Shared Data Queue is accessible in the initialize() block.

info("Create queueA with life time of 120 minutes");
sharedData.createQueue("queueA", 120);

In the run() block the following code will remove the first element of the Queue and write an informational message with the current username Value to the Result window.

info("Get and remove the current user from the head of queueA");
String pollValue1 = (String) sharedData.pollFirst("queueA");
info("The first user in queueA was currently: '" + pollValue1 + "' " + pollValue1.getClass() + " length " + pollValue1.length() );

References

Oracle Functional Testing OpenScript User's Guide Version 9.20 [E15488-05]
Chapter 17 Using the Shared Data Module
http://download.oracle.com/otn/nt/apptesting/oats-docs-9.21.0030.zip

Oracle Fusion Middleware Oracle WebLogic Server Administration Console Online Help 11g Release 1 (10.3.4) [E13952-04]
Administration Console Online Help - Manage users and groups
http://download.oracle.com/docs/cd/E17904_01/apirefs.1111/e13952/taskhelp/security/ManageUsersAndGroups.htm