I have a very basic load test question.

I am running a load test using VSTS 2008 and I have test rig with controller + 10 agents. This load test is against a SharePoint farm I have. My goal of the load test is to find out the resource utilization on web+app+db tiers of my farm for any given load scenario. An example of a load scenario is

Usage profile: Average collaboration (as defined by SCCP)

User Load: 500 (using step load pattern=a step of 50 every 2 mins and a warm up time of 2mins for every step)

Think time: 0

Load duration: 8hrs

Now, the question is: Is it fair to expect that metrics like Requests/sec, %processor time on web front end / App / DB, Test/sec, and etc become flat or enter a steady state at one point in time during the load test. Like I said, the goal is not to create a bottleneck but to only measure the utilization of resources by the above load profile.

I am asking this question because I see something different. At one point in the load test, requests/sec becomes more or less flat. But processor utilization on the web/DB servers keeps increasing. After digging through the data a bit, I see that "tests running" counter also steadily increased over time. So, if I run the load test for more than 8hrs, %processor may go up further. This way, I don't know what to consider as the load excreted by the load profile.

What does this "tests running" counter really signify? How is this different from tests/sec?

Another question is: how can I find out why "tests running" counter shows an increase overtime?

Thanks for your time