Originally posted on: http://geekswithblogs.net/TarunArora/archive/2013/06/30/cloud-based-load-testing-using-tf-service-amp-vs-2013.aspx

One of the new features announced as part of the Visual Studio 2013 Ultimate Preview is ‘Cloud Based Load Testing’. In this blog post I’ll walk you through,

1. What is Cloud Based Load Testing?
2. How have I been using this feature? – Success story!
3. Where can you find more resources on this feature?

What is Cloud Based Load Testing?

It goes without saying that performance testing your application not only gives you the confidence that the application will work under heavy levels of stress but also gives you the ability to test how scalable the architecture of your application is. It is important to know how much is too much for your application!

Working with various clients in the industry I have realized that the biggest barriers in Load Testing & Performance Testing adoption are,

• High infrastructure and administration cost that comes with this phase of testing
• Time taken to procure & set up the test infrastructure
• Finding use for this infrastructure investment after completion of testing

Is cloud the answer?

• 100% Visual Studio Compatible
• Scalable and Realistic
• Start testing in < 2 minutes
• Intuitive
• Pay only for what you need
• Use existing on premise tests on cloud

There are a lot of vendors out there offering Cloud Based Load Testing, to name a few,

• Load Storm
• Soasta
• Blaze Meter
• Blitz
• And others…

The question you may want to ask is, why should you go with Microsoft’s Cloud based Load Test offering.

If you are a Microsoft shop or already have investments in Microsoft technologies, you’ll see great benefit in the natural integration this offers with existing Microsoft products such as Visual Studio and Windows Azure. For example, your existing Web tests authored in Visual Studio 2010 or Visual Studio 2012 will run on the cloud without requiring any modifications what so ever. Microsoft’s cloud test rig also supports API based testing, for example, if you are building a WPF application which consumes WCF services, you can write unit tests to invoke the WCF service, these tests can be run on the cloud test rig and loaded with ‘N’ concurrent users for performance testing. If you have your assets already hosted in the Azure and possibly in the same data centre as the Cloud test rig, your Azure app will not incur a usage cost because of the generated traffic since the traffic is coming from the same data centre. The licensing or pricing information on Microsoft’s cloud based Load test service is yet to be announced, but I would expect this to be priced attractively to match the market competition.

The only additional configuration required for running load tests on Microsoft Cloud based Load Tests service is to select the Test run location as Run tests using Visual Studio Team Foundation Service,

How have I been using Microsoft’s Cloud based Load Test Service?

I have been part of the Microsoft Cloud Based Load Test Service advisory council for the last 7 months. This gave the opportunity to see the product shape up from concept to working solution. I was also the first person outside of Microsoft to try this offering out. This gave me the opportunity to test real world application at various clients using the Microsoft Load Test Service and provide real world feedback to the Microsoft product team.

One of the most recent systems I tested using the Load Test Service has been an insurance quote generation engine. This insurance quote generation engine is,

• hosted in Windows Azure
• expected to get quote requests from across the globe
• expected to handle 5 Million quote requests in a day (not clear how this load will be distributed across the day)

There was no way, I could simulate such kind of load from on premise without standing up additional hardware. But Microsoft’s Cloud based Load Test service allowed me to test my key performance testing scenarios, i.e. Simulate expected Load, Endurance Testing, Threshold Testing and Testing for Latency.

Simulating expected load: approach to devising a load pattern

My approach to devising a load test pattern has been to run the test scenario with 1 user to figure out the response time. Then work out how many users are required to reach the target load.

So, for example, to invoke 1 quote from the quote engine software takes 0.5 seconds.

Now if you do the math,

  1 quote request by 1 user = 0.5 seconds

  quotes generated by 1 user in 24 hour = 1 * (((2 * 60) * 60) * 24) = 172,800

  quotes generated by 30 users in 24 hours = 172,800 * 30 =  5,184,000

This was a very simple example, if your application requires more concurrent users to test scenario’s such as caching, etc then you can devise your own load pattern, some examples of load test patterns can be found here. 

Endurance Testing

To test for endurance, I loaded the quote generation engine with an expected fixed user load and ran the test for very long duration such as over 48 hours and observed the affect of the long running test on the Azure infrastructure. Currently Microsoft Load Test service does not support metrics from the machine under test. I used Azure diagnostics to begin with, but later started using Cerebrata Azure Diagnostics Manager to capture the metrics of the machine under test.

Threshold Testing

To figure out how much user load the application could cope with before falling on its belly, I opted to step load the quote generation engine by incrementing user load with different variations of incremental user load per minute till the application crashed out and forced an IIS reset.

Testing for Latency

Currently the Microsoft Load Test service does not support generating geographically distributed load, I however, deployed the insurance quote generation engine in different Azure data centres and ran the same set of performance tests to measure for latency. Because I could compare load test results from different runs by exporting the results to excel (this feature is provided out of the box right from Visual Studio 2010) I could see the different in response times.

More resources on Microsoft Cloud based Load Test Service

A few important links to get you started,

• Download Visual Studio Ultimate 2013 Preview
• Getting started guide for load testing using Team Foundation Service
• Troubleshooting guide for FAQs and known issues
• Team Foundation Service forum for questions and support
• Detailed demo and presentation (link to Tech-Ed session recording)
• Detailed demo and presentation (link to Build session recording)

There a few limits on the usage of Microsoft Cloud based Load Test service that you can read about here. If you have any feedback on Microsoft Cloud based Load Test service, feel free to share it with the product team via the Visual Studio User Voice forum.

I hope you found this useful. Thank you for taking the time out and reading this blog post. If you enjoyed the post, remember to subscribe to http://feeds.feedburner.com/TarunArora. Stay tuned!