A few days ago I had a curious thought: With all these different technologies that
the ASP.NET stack has to offer, what's
the most efficient technology overall to return data for a server request? When I started this it was mere curiosity rather than a real practical need or result. Different tools are used for different problems and so performance differences are to be expected. But still I was curious to see how
the various technologies performed relative to each just for raw throughput of
the request getting to
the endpoint and back out to
the client with as little processing in
the actual endpoint logic as possible (aka Hello World!). I want to clarify that this is merely an informal test for my own curiosity and I'm sharing
the results and process here because I thought it was interesting. It's been a long while since I've done any sort of perf testing on ASP.NET, mainly because I've not had extremely heavy load requirements and because overall ASP.NET performs very well even for fairly high loads so that often it's not that critical to test load performance. This post is not meant to make a point or even come to a conclusion which tech is better, but just to act as a reference to help understand some of
the differences in perf and give a starting point to play around with this yourself. I've included
the code for this simple project, so you can play with it and maybe add a few additional tests for different things if you like. Source Code on GitHub I looked at this data for these technologies: ASP.NET Web API ASP.NET MVC WebForms ASP.NET WebPages ASMX AJAX Services (couldn't get AJAX/JSON to run on IIS8 ) WCF Rest Raw ASP.NET HttpHandlers It's quite a mixed bag, of course and
the technologies target different types of development. What started out as mere curiosity turned into a bit of a head scratcher as
the results were sometimes surprising. What I describe here is more to satisfy my curiosity more than anything and I thought it interesting enough to discuss on
the blog :-) First test: Raw Throughput
The first thing I did is test raw throughput for
the various technologies. This is
the least practical test of course since you're unlikely to ever create
the equivalent of a 'Hello World' request in a real life application.
The idea here is to measure how much time a 'NOP' request takes to return data to
the client. So for this request I create
the simplest Hello World request that I could come up for each tech. Http Handler
The first is
the lowest level approach which is an HTTP handler. public class Handler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "text/plain";
context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
}
public bool IsReusable
{
get
{
return true;
}
}
}
WebForms
Next I added a couple of ASPX pages - one using CodeBehind and one using only a markup page.
The CodeBehind page simple does this in CodeBehind without any markup in
the ASPX page: public partial class HelloWorld_CodeBehind : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
Response.Write("Hello World. Time is: " + DateTime.Now.ToString() );
Response.End();
}
}
while
the Markup page only contains some static output via an expression:<%@ Page Language="C#" AutoEventWireup="false"
CodeBehind="HelloWorld_Markup.aspx.cs"
Inherits="AspNetFrameworksPerformance.HelloWorld_Markup" %>
Hello World. Time is <%= DateTime.Now %>
ASP.NET WebPages
WebPages is
the freestanding Razor implementation of ASP.NET. Here's
the simple HelloWorld.cshtml page:Hello World @DateTime.Now
WCF REST
WCF REST was
the token REST implementation for ASP.NET before WebAPI and
the inbetween step from ASP.NET AJAX. I'd like to forget that this technology was ever considered for production use, but I'll include it here. Here's an OperationContract class:
[ServiceContract(Namespace = "")]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class WcfService
{
[OperationContract]
[WebGet]
public Stream HelloWorld()
{
var data = Encoding.Unicode.GetBytes("Hello World" + DateTime.Now.ToString());
var ms = new MemoryStream(data);
// Add your operation implementation here
return ms;
}
}
WCF REST can return arbitrary results by returning a Stream object and a content type.
The code above turns
the string result into a stream and returns that back to
the client.
ASP.NET AJAX (ASMX Services)
I also wanted to test ASP.NET AJAX services because prior to WebAPI this is probably still
the most widely used AJAX technology for
the ASP.NET stack today.
Unfortunately I was completely unable to get this running on my Windows 8 machine. Visual Studio 2012 removed adding of ASP.NET AJAX services, and when I tried to manually add
the service and configure
the script handler references it simply did not work - I always got a SOAP response for GET and POST operations. No matter what I tried I always ended up getting XML results even when explicitly adding
the ScriptHandler. So, I didn't test this (but
the code is there - you might be able to test this on a Windows 7 box).
ASP.NET MVC
Next up is probably
the most popular ASP.NET technology at
the moment: MVC. Here's
the small controller: public class MvcPerformanceController : Controller
{
public ActionResult Index()
{
return View();
}
public ActionResult HelloWorldCode()
{
return new ContentResult()
{ Content = "Hello World. Time is: " + DateTime.Now.ToString() };
}
}
ASP.NET WebAPI
Next up is WebAPI which looks kind of similar to MVC. Except here I have to use a StringContent result to return
the response: public class WebApiPerformanceController : ApiController
{
[HttpGet]
public HttpResponseMessage HelloWorldCode()
{
return new HttpResponseMessage()
{ Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
}
}
Testing
Take a minute to think about each of
the technologies… and take a guess which you think is most efficient in raw throughput.
The fastest should be pretty obvious, but
the others - maybe not so much.
The testing I did is pretty informal since it was mainly to satisfy my curiosity - here's how I did this: I used Apache Bench (ab.exe) from a full Apache HTTP installation to run and log
the test results of hitting
the server. ab.exe is a small executable that lets you hit a URL repeatedly and provides counter information about
the number of requests, requests per second etc. ab.exe and
the batch file are located in
the \LoadTests folder of
the project.
An ab.exe command line looks like this:
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld
which hits
the specified URL 100,000 times with a load factor of 20 concurrent requests. This results in output like this:
It's a great way to get a quick and dirty performance summary. Run it a few times to make sure there's not a large amount of varience. You might also want to do an IISRESET to clear
the Web Server. Just make sure you do a short test run to warm up
the server first - otherwise your first run is likely to be skewed downwards. ab.exe also allows you to specify headers and provide POST data and many other things if you want to get a little more fancy. Here all tests are GET requests to keep it simple.
I ran each test:
100,000 iterations
Load factor of 20 concurrent connections
IISReset before starting
A short warm up run for API and MVC to make sure startup cost is mitigated
Here is
the batch file I used for
the test:
IISRESET
REM make sure you add
REM C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin
REM to your path so ab.exe can be found
REM Warm up
ab.exe -n100 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJsonab.exe -n100 -c20 http://localhost/aspnetperf/api/HelloWorldJson
ab.exe -n100 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld
ab.exe -n100000 -c20 http://localhost/aspnetperf/handler.ashx > handler.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_CodeBehind.aspx > AspxCodeBehind.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_Markup.aspx > AspxMarkup.txt
ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld > Wcf.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldCode > Mvc.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld > WebApi.txt
I ran each of these tests 3 times and took
the average score for Requests/second, with
the machine otherwise idle. I did see a bit of variance when running many tests but
the values used here are
the medians. Part of this has to do with
the fact I ran
the tests on my local machine - result would probably more consistent running
the load test on a separate machine hitting across
the network.
I ran these tests locally on my laptop which is a Dell XPS with quad core Sandibridge I7-2720QM @ 2.20ghz and a fast SSD drive on Windows 8. CPU load during tests ran to about 70% max across all 4 cores (IOW, it wasn't overloading
the machine). Ideally you can try running these tests on a separate machine hitting
the local machine. If I remember correctly IIS 7 and 8 on client OSs don't throttle so
the performance here should be
Results
Ok, let's cut straight to
the chase. Below are
the results from
the tests…
It's not surprising that
the handler was fastest. But it was a bit surprising to me that
the next fastest was WebForms and especially Web Forms with markup over a CodeBehind page. WebPages also fared fairly well. MVC and WebAPI are a little slower and
the slowest by far is WCF REST (which again I find surprising).
As mentioned at
the start
the raw throughput tests are not overly practical as they don't test scripting performance for
the HTML generation engines or serialization performances of
the data engines. All it really does is give you an idea of
the raw throughput for
the technology from time of request to reaching
the endpoint and returning minimal text data back to
the client which indicates full round trip performance.
But it's still interesting to see that Web Forms performs better in throughput than either MVC, WebAPI or WebPages. It'd be interesting to try this with a few pages that actually have some parsing logic on it, but that's beyond
the scope of this throughput test.
But what's also amazing about this test is
the sheer amount of traffic that a laptop computer is handling. Even
the slowest tech managed 5700 requests a second, which is one hell of a lot of requests if you extrapolate that out over a 24 hour period. Remember these are not static pages, but dynamic requests that are being served.
Another test - JSON Data Service Results
The second test I used a JSON result from several of
the technologies. I didn't bother running WebForms and WebPages through this test since that doesn't make a ton of sense to return data from
the them (OTOH, returning text from
the APIs didn't make a ton of sense either :-)
In these tests I have a small
Person class that gets serialized and then returned to
the client.
The Person class looks like this: public class
Person
{
public Person()
{
Id = 10;
Name = "Rick";
Entered = DateTime.Now;
}
public int Id { get; set; }
public string Name { get; set; }
public DateTime Entered { get; set; }
}
Here are
the updated handler classes that use Person:
Handler public class Handler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
var action = context.Request.QueryString["action"];
if (action == "json")
JsonRequest(context);
else
TextRequest(context);
}
public void TextRequest(HttpContext context)
{
context.Response.ContentType = "text/plain";
context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
}
public void JsonRequest(HttpContext context)
{
var json = JsonConvert.SerializeObject(new Person(), Formatting.None);
context.Response.ContentType = "application/json";
context.Response.Write(json);
}
public bool IsReusable
{
get
{
return true;
}
}
}
This code adds a little logic to check for a action query string and route
the request to an optional JSON result method. To generate JSON, I'm using
the same JSON.NET serializer (JsonConvert.SerializeObject) used in Web API to create
the JSON response.
WCF REST
[ServiceContract(Namespace = "")]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class WcfService
{
[OperationContract]
[WebGet]
public Stream HelloWorld()
{
var data = Encoding.Unicode.GetBytes("Hello World " + DateTime.Now.ToString());
var ms = new MemoryStream(data);
// Add your operation implementation here
return ms;
}
[OperationContract]
[WebGet(ResponseFormat=WebMessageFormat.Json,BodyStyle=WebMessageBodyStyle.WrappedRequest)]
public
Person HelloWorldJson()
{
// Add your operation implementation here
return new Person();
}
}
For WCF REST all I have to do is add a method with
the Person result type.
ASP.NET MVC public class MvcPerformanceController : Controller
{
//
// GET: /MvcPerformance/
public ActionResult Index()
{
return View();
}
public ActionResult HelloWorldCode()
{
return new ContentResult()
{ Content = "Hello World. Time is: " + DateTime.Now.ToString() };
}
public JsonResult HelloWorldJson()
{
return Json(new Person(), JsonRequestBehavior.AllowGet);
}
}
For MVC all I have to do for a JSON response is return a JSON result. ASP.NET internally uses JavaScriptSerializer.
ASP.NET WebAPI public class WebApiPerformanceController : ApiController
{
[HttpGet]
public HttpResponseMessage HelloWorldCode()
{
return new HttpResponseMessage()
{ Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
}
[HttpGet]
public
Person HelloWorldJson()
{
return new Person();
}
[HttpGet]
public HttpResponseMessage HelloWorldJson2()
{
var response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new ObjectContent<Person>(new Person(),
GlobalConfiguration.Configuration.Formatters.JsonFormatter);
return response;
}
}
Testing and Results
To run these data requests I used
the following ab.exe commands:REM JSON RESPONSES
ab.exe -n100000 -c20 http://localhost/aspnetperf/Handler.ashx?action=json > HandlerJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJson > MvcJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorldJson > WebApiJson.txt
ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorldJson > WcfJson.txt
The results from this test run are a bit interesting in that
the WebAPI test improved performance significantly over returning plain string content.
Here are
the results:
The performance for each technology drops a little bit except for WebAPI which is up quite a bit! From this test it appears that WebAPI is actually significantly better performing returning a JSON response, rather than a plain string response.
Snag with Apache Benchmark and 'Length Failures'
I ran into a little snag with Apache Benchmark, which was reporting failures for my Web API requests when serializing. As
the graph shows performance improved significantly from with JSON results from 5580 to 6530 or so which is a 15% improvement (while all others slowed down by 3-8%).
However, I was skeptical at first because
the WebAPI test reports showed a bunch of errors on about 10% of
the requests. Check out this report:
Notice
the Failed Request count. What
the hey? Is WebAPI failing on roughly 10% of requests when sending JSON?
Turns out: No it's not! But it took some sleuthing to figure out why it reports these failures. At first I thought that Web API was failing, and so to make sure I re-ran
the test with Fiddler attached and runiisning
the ab.exe test by using
the -X switch:
ab.exe -n100 -c10 -X localhost:8888 http://localhost/aspnetperf/api/HelloWorldJson
which showed that indeed all requests where returning proper HTTP 200 results with full content.
However ab.exe was reporting
the errors. After some closer inspection it turned out that
the dates varying in size altered
the response length in dynamic output.
For example: these two results:
{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.841926-10:00"}
{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.8519262-10:00"}
are different in length for
the number which results in 68 and 69 bytes respectively.
The same URL produces different result lengths which is what ab.exe reports. I didn't notice at first bit
the same is happening when running
the ASHX handler with JSON.NET result since it uses
the same serializer that varies
the milliseconds.
Moral: You can typically ignore Length failures in Apache Benchmark and when in doubt check
the actual output with Fiddler. Note that
the other failure values are accurate though.
Another interesting Side Note: Perf drops over Time
As I was running these tests repeatedly I was finding that performance steadily dropped from a startup peak to a 10-15% lower stable level. IOW, with Web API I'd start out with around 6500 req/sec and in subsequent runs it keeps dropping until it would stabalize somewhere around 5900 req/sec occasionally jumping lower. For these tests this is why I did
the IIS RESET and warm up for individual tests. This is a little puzzling. Looking at Process Monitor while
the test are running memory very quickly levels out as do handles and threads, on
the first test run. Subsequent runs everything stays stable, but
the performance starts going downwards. This applies to all
the technologies - Handlers, Web Forms, MVC, Web API - curious to see if others test this and see similar results.
Doing an IISRESET then resets everything and performance starts off at peak again…
Summary
As I stated at
the outset, these were informal to satiate my curiosity not to prove that any technology is better or even faster than another. While there clearly are differences in performance
the differences (other than WCF REST which was by far
the slowest and
the raw handler which was by far
the highest) are relatively minor, so there is no need to feel that any one technology is a runaway standout in raw performance. Choosing a technology is about more than pure performance but also about
the adequateness for
the job and
the easy of implementation.
The strengths of each technology will make for any minor performance difference we see in these tests.
However, to me it's important to get an occasional reality check and compare where new technologies are heading. Often times old stuff that's been optimized and designed for a time of less horse power can utterly blow
the doors off newer tech and simple checks like this let you compare. Luckily we're seeing that much of
the new stuff performs well even in V1.0 which is great.
To me it was very interesting to see Web API perform relatively badly with plain string content, which originally led me to think that Web API might not be properly optimized just yet. For those that caught my Tweets late last week regarding WebAPI's slow responses was with String content which is in fact considerably slower. Luckily where it counts with serialized JSON and XML WebAPI actually performs better. But I do wonder what would make generic string content slower than serialized code?
This stresses another point: Don't take a single test as
the final gospel and don't extrapolate out from a single set of tests. Certainly Twitter can make you feel like a fool when you post something immediate that hasn't been fleshed out a little more <blush>. Egg on my face. As a result I ended up screwing around with this for a few hours today to compare different scenarios. Well worth
the time…
I hope you found this useful, if not for
the results, maybe for
the process of quickly testing a few requests for performance and charting out a comparison. Now onwards with more serious stuff…
Resources
Source Code on GitHub
Apache HTTP Server Project (ab.exe is part of
the binary distribution)© Rick Strahl, West Wind Technologies, 2005-2012Posted in ASP.NET Web Api
Tweet
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
(function() {
var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true;
po.src = 'https://apis.google.com/js/plusone.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s);
})();
