A few days ago I had a curious thought: With all these different technologies that the ASP.NET stack has
to offer, what's the most efficient technology overall
to return data for a server request? When I started this it was mere curiosity rather than a real practical need or result. Different tools are used for different problems and so performance differences are
to be expected. But still I was curious
to see how the various technologies performed relative
to each just for raw throughput of the request getting
to the endpoint and back out
to the client with as little processing in the actual endpoint logic as possible (aka Hello World!). I want
to clarify that this is merely an informal test for my own curiosity and I'm sharing the results and process here because I thought it was interesting. It's been a long while since I've done any sort of perf testing on ASP.NET, mainly because I've not had extremely heavy load requirements and because overall ASP.NET performs very well even for fairly high loads so that often it's not that critical
to test load performance. This post is not meant
to make a point or even come
to a conclusion which tech is better, but just
to act as a reference
to help understand some of the differences in perf and give a starting point
to play around with this yourself. I've included the code for this simple project, so you can play with it and maybe add a few additional tests for different things if you like. Source Code on GitHub I looked at this data for these technologies: ASP.NET Web API ASP.NET MVC WebForms ASP.NET WebPages ASMX AJAX Services (couldn't get AJAX/JSON
to run on IIS8 ) WCF Rest Raw ASP.NET HttpHandlers It's quite a mixed bag, of course and the technologies target different types of development. What started out as mere curiosity turned into a bit of a head scratcher as the results were sometimes surprising. What I describe here is more
to satisfy my curiosity more than anything and I thought it interesting enough
to discuss on the blog :-) First test: Raw Throughput The first thing I did is test raw throughput for the various technologies. This is the least practical test of course since you're unlikely
to ever create the equivalent of a 'Hello World' request in a real life application. The idea here is
to measure how much time a 'NOP' request takes
to return data
to the client. So for this request I create the simplest Hello World request that I could come up for each tech. Http Handler The first is the lowest level approach which is an HTTP handler. public class Handler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "text/plain";
context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
}
public bool IsReusable
{
get
{
return true;
}
}
}
WebForms
Next I added a couple of ASPX pages - one using CodeBehind and one using only a markup page.
The CodeBehind page simple does this in CodeBehind without any markup in the ASPX page: public partial class HelloWorld_CodeBehind : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
Response.Write("Hello World. Time is: " + DateTime.Now.ToString() );
Response.End();
}
}
while the Markup page only contains some static output via an expression:<%@ Page Language="C#" AutoEventWireup="false"
CodeBehind="HelloWorld_Markup.aspx.cs"
Inherits="AspNetFrameworksPerformance.HelloWorld_Markup" %>
Hello World. Time is <%= DateTime.Now %>
ASP.NET WebPages
WebPages is the freestanding Razor implementation of ASP.NET. Here's the simple HelloWorld.cshtml page:Hello World @DateTime.Now
WCF REST
WCF REST was the token REST implementation for ASP.NET before WebAPI and the inbetween step from ASP.NET AJAX. I'd like
to forget that this technology was ever considered for production use, but I'll include it here. Here's an OperationContract class:
[ServiceContract(Namespace = "")]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class WcfService
{
[OperationContract]
[WebGet]
public Stream HelloWorld()
{
var data = Encoding.Unicode.GetBytes("Hello World" + DateTime.Now.ToString());
var ms = new MemoryStream(data);
// Add your operation implementation here
return ms;
}
}
WCF REST can return arbitrary results by returning a Stream
object and a content type. The code above turns the
string result into a stream and returns that back
to the client.
ASP.NET AJAX (ASMX Services)
I also wanted
to test ASP.NET AJAX services because prior
to WebAPI this is probably still the most widely used AJAX technology for the ASP.NET stack today.
Unfortunately I was completely unable
to get this running on my Windows 8 machine. Visual Studio 2012 removed adding of ASP.NET AJAX services, and when I tried
to manually add the service and configure the script handler references it simply did not work - I always got a SOAP response for GET and POST operations. No matter what I tried I always ended up getting XML results even when explicitly adding the ScriptHandler. So, I didn't test this (but the code is there - you might be able
to test this on a Windows 7 box).
ASP.NET MVC
Next up is probably the most popular ASP.NET technology at the moment: MVC. Here's the small controller: public class MvcPerformanceController : Controller
{
public ActionResult Index()
{
return View();
}
public ActionResult HelloWorldCode()
{
return new ContentResult()
{ Content = "Hello World. Time is: " + DateTime.Now.ToString() };
}
}
ASP.NET WebAPI
Next up is WebAPI which looks kind of similar
to MVC. Except here I have
to use a StringContent result
to return the response: public class WebApiPerformanceController : ApiController
{
[HttpGet]
public HttpResponseMessage HelloWorldCode()
{
return new HttpResponseMessage()
{ Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
}
}
Testing
Take a minute
to think about each of the technologies… and take a guess which you think is most efficient in raw throughput. The fastest should be pretty obvious, but the others - maybe not so much.
The testing I did is pretty informal since it was mainly
to satisfy my curiosity - here's how I did this: I used Apache Bench (ab.exe) from a full Apache HTTP installation
to run and log the test results of hitting the server. ab.exe is a small executable that lets you hit a URL repeatedly and provides counter information about the number of requests, requests per second etc. ab.exe and the batch file are located in the \LoadTests folder of the project.
An ab.exe command line looks like this:
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld
which hits the specified URL 100,000 times with a load factor of 20 concurrent requests. This results in output like this:
It's a great way
to get a quick and dirty performance summary. Run it a few times
to make sure there's not a large amount of varience. You might also want
to do an IISRESET
to clear the Web Server. Just make sure you do a short test run
to warm up the server first - otherwise your first run is likely
to be skewed downwards. ab.exe also allows you
to specify headers and provide POST data and many other things if you want
to get a little more fancy. Here all tests are GET requests
to keep it simple.
I ran each test:
100,000 iterations
Load factor of 20 concurrent connections
IISReset before starting
A short warm up run for API and MVC
to make sure startup cost is mitigated
Here is the batch file I used for the test:
IISRESET
REM make sure you add
REM C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin
REM
to your path so ab.exe can be found
REM Warm up
ab.exe -n100 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJsonab.exe -n100 -c20 http://localhost/aspnetperf/api/HelloWorldJson
ab.exe -n100 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld
ab.exe -n100000 -c20 http://localhost/aspnetperf/handler.ashx > handler.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_CodeBehind.aspx > AspxCodeBehind.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_Markup.aspx > AspxMarkup.txt
ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld > Wcf.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldCode > Mvc.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld > WebApi.txt
I ran each of these tests 3 times and took the average score for Requests/second, with the machine otherwise idle. I did see a bit of variance when running many tests but the values used here are the medians. Part of this has
to do with the fact I ran the tests on my local machine - result would probably more consistent running the load test on a separate machine hitting across the network.
I ran these tests locally on my laptop which is a Dell XPS with quad core Sandibridge I7-2720QM @ 2.20ghz and a fast SSD drive on Windows 8. CPU load during tests ran
to about 70% max across all 4 cores (IOW, it wasn't overloading the machine). Ideally you can try running these tests on a separate machine hitting the local machine. If I remember correctly IIS 7 and 8 on client OSs don't throttle so the performance here should be
Results
Ok, let's cut straight
to the chase. Below are the results from the tests…
It's not surprising that the handler was fastest. But it was a bit surprising
to me that the next fastest was WebForms and especially Web Forms with markup over a CodeBehind page. WebPages also fared fairly well. MVC and WebAPI are a little slower and the slowest by far is WCF REST (which again I find surprising).
As mentioned at the start the raw throughput tests are not overly practical as they don't test scripting performance for the HTML generation engines or serialization performances of the data engines. All it really does is give you an idea of the raw throughput for the technology from time of request
to reaching the endpoint and returning minimal text data back
to the client which indicates full round trip performance.
But it's still interesting
to see that Web Forms performs better in throughput than either MVC, WebAPI or WebPages. It'd be interesting
to try this with a few pages that actually have some parsing logic on it, but that's beyond the scope of this throughput test.
But what's also amazing about this test is the sheer amount of traffic that a laptop computer is handling. Even the slowest tech managed 5700 requests a second, which is one hell of a lot of requests if you extrapolate that out over a 24 hour period. Remember these are not static pages, but dynamic requests that are being served.
Another test - JSON Data Service Results
The second test I used a JSON result from several of the technologies. I didn't bother running WebForms and WebPages through this test since that doesn't make a ton of sense
to return data from the them (OTOH, returning text from the APIs didn't make a ton of sense either :-)
In these tests I have a small Person class that gets serialized and then returned
to the client. The Person class looks like this: public class Person
{
public Person()
{
Id = 10;
Name = "Rick";
Entered = DateTime.Now;
}
public int Id { get; set; }
public
string Name { get; set; }
public DateTime Entered { get; set; }
}
Here are the updated handler classes that use Person:
Handler public class Handler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
var action = context.Request.QueryString["action"];
if (action == "json")
JsonRequest(context);
else
TextRequest(context);
}
public void TextRequest(HttpContext context)
{
context.Response.ContentType = "text/plain";
context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
}
public void JsonRequest(HttpContext context)
{
var json = JsonConvert.SerializeObject(new Person(), Formatting.None);
context.Response.ContentType = "application/json";
context.Response.Write(json);
}
public bool IsReusable
{
get
{
return true;
}
}
}
This code adds a little logic
to check for a action query
string and route the request
to an optional JSON result method.
To generate JSON, I'm using the same JSON.NET serializer (JsonConvert.SerializeObject) used in Web API
to create the JSON response.
WCF REST
[ServiceContract(Namespace = "")]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class WcfService
{
[OperationContract]
[WebGet]
public Stream HelloWorld()
{
var data = Encoding.Unicode.GetBytes("Hello World " + DateTime.Now.ToString());
var ms = new MemoryStream(data);
// Add your operation implementation here
return ms;
}
[OperationContract]
[WebGet(ResponseFormat=WebMessageFormat.Json,BodyStyle=WebMessageBodyStyle.WrappedRequest)]
public Person HelloWorldJson()
{
// Add your operation implementation here
return new Person();
}
}
For WCF REST all I have
to do is add a method with the Person result type.
ASP.NET MVC public class MvcPerformanceController : Controller
{
//
// GET: /MvcPerformance/
public ActionResult Index()
{
return View();
}
public ActionResult HelloWorldCode()
{
return new ContentResult()
{ Content = "Hello World. Time is: " + DateTime.Now.ToString() };
}
public JsonResult HelloWorldJson()
{
return Json(new Person(), JsonRequestBehavior.AllowGet);
}
}
For MVC all I have
to do for a JSON response is return a JSON result. ASP.NET internally uses JavaScriptSerializer.
ASP.NET WebAPI public class WebApiPerformanceController : ApiController
{
[HttpGet]
public HttpResponseMessage HelloWorldCode()
{
return new HttpResponseMessage()
{ Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
}
[HttpGet]
public Person HelloWorldJson()
{
return new Person();
}
[HttpGet]
public HttpResponseMessage HelloWorldJson2()
{
var response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new ObjectContent<Person>(new Person(),
GlobalConfiguration.Configuration.Formatters.JsonFormatter);
return response;
}
}
Testing and Results
To run these data requests I used the following ab.exe commands:REM JSON RESPONSES
ab.exe -n100000 -c20 http://localhost/aspnetperf/Handler.ashx?action=json > HandlerJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJson > MvcJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorldJson > WebApiJson.txt
ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorldJson > WcfJson.txt
The results from this test run are a bit interesting in that the WebAPI test improved performance significantly over returning plain
string content.
Here are the results:
The performance for each technology drops a little bit except for WebAPI which is up quite a bit! From this test it appears that WebAPI is actually significantly better performing returning a JSON response, rather than a plain
string response.
Snag with Apache Benchmark and 'Length Failures'
I ran into a little snag with Apache Benchmark, which was reporting failures for my Web API requests when serializing. As the graph shows performance improved significantly from with JSON results from 5580
to 6530 or so which is a 15% improvement (while all others slowed down by 3-8%).
However, I was skeptical at first because the WebAPI test reports showed a bunch of errors on about 10% of the requests. Check out this report:
Notice the Failed Request count. What the hey? Is WebAPI failing on roughly 10% of requests when sending JSON?
Turns out: No it's not! But it took some sleuthing
to figure out why it reports these failures. At first I thought that Web API was failing, and so
to make sure I re-ran the test with Fiddler attached and runiisning the ab.exe test by using the -X switch:
ab.exe -n100 -c10 -X localhost:8888 http://localhost/aspnetperf/api/HelloWorldJson
which showed that indeed all requests where returning proper HTTP 200 results with full content.
However ab.exe was reporting the errors. After some closer inspection it turned out that the dates varying in size altered the response length in dynamic output.
For example: these two results:
{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.841926-10:00"}
{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.8519262-10:00"}
are different in length for the number which results in 68 and 69 bytes respectively. The same URL produces different result lengths which is what ab.exe reports. I didn't notice at first bit the same is happening when running the ASHX handler with JSON.NET result since it uses the same serializer that varies the milliseconds.
Moral: You can typically ignore Length failures in Apache Benchmark and when in doubt check the actual output with Fiddler. Note that the other failure values are accurate though.
Another interesting Side Note: Perf drops over Time
As I was running these tests repeatedly I was finding that performance steadily dropped from a startup peak
to a 10-15% lower stable level. IOW, with Web API I'd start out with around 6500 req/sec and in subsequent runs it keeps dropping until it would stabalize somewhere around 5900 req/sec occasionally jumping lower. For these tests this is why I did the IIS RESET and warm up for individual tests. This is a little puzzling. Looking at Process Monitor while the test are running memory very quickly levels out as do handles and threads, on the first test run. Subsequent runs everything stays stable, but the performance starts going downwards. This applies
to all the technologies - Handlers, Web Forms, MVC, Web API - curious
to see if others test this and see similar results.
Doing an IISRESET then resets everything and performance starts off at peak again…
Summary
As I stated at the outset, these were informal
to satiate my curiosity not
to prove that any technology is better or even faster than another. While there clearly are differences in performance the differences (other than WCF REST which was by far the slowest and the raw handler which was by far the highest) are relatively minor, so there is no need
to feel that any one technology is a runaway standout in raw performance. Choosing a technology is about more than pure performance but also about the adequateness for the job and the easy of implementation. The strengths of each technology will make for any minor performance difference we see in these tests.
However,
to me it's important
to get an occasional reality check and compare where new technologies are heading. Often times old stuff that's been optimized and designed for a time of less horse power can utterly blow the doors off newer tech and simple checks like this let you compare. Luckily we're seeing that much of the new stuff performs well even in V1.0 which is great.
To me it was very interesting
to see Web API perform relatively badly with plain
string content, which originally led me
to think that Web API might not be properly optimized just yet. For those that caught my Tweets late last week regarding WebAPI's slow responses was with
String content which is in fact considerably slower. Luckily where it counts with serialized JSON and XML WebAPI actually performs better. But I do wonder what would make generic
string content slower than serialized code?
This stresses another point: Don't take a single test as the final gospel and don't extrapolate out from a single set of tests. Certainly Twitter can make you feel like a fool when you post something immediate that hasn't been fleshed out a little more <blush>. Egg on my face. As a result I ended up screwing around with this for a few hours today
to compare different scenarios. Well worth the time…
I hope you found this useful, if not for the results, maybe for the process of quickly testing a few requests for performance and charting out a comparison. Now onwards with more serious stuff…
Resources
Source Code on GitHub
Apache HTTP Server Project (ab.exe is part of the binary distribution)© Rick Strahl, West Wind Technologies, 2005-2012Posted in ASP.NET Web Api
Tweet
!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
(function() {
var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true;
po.src = 'https://apis.google.com/js/plusone.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s);
})();
