Background
One of the clients I work with had been experiencing some issues for a while surrounding web service timeouts. It's been a little challenging to work through the problems due to limitations in the diagnostic information available from one of the applications, but I learned some interesting things while troubleshooting the problem which don't seem to have been discussed much in the community so I thought I'd share my findings.
In the scenario we have BizTalk trying to make calls to a .net web service which was exposed as a WSE 2 endpoint. In the process BizTalk will try to make a large number of concurrent web service calls to the application, and the backend application has more than enough infrastructure and capability to handle the load.
We have configured the <ConnectionManagement>
section of the BizTalk configuration file to support up to 100 concurrent connections from each of our 2 BizTalk send servers to the web servers of the application.
The problem we were facing was that the BizTalk side was reporting a significant number of timeouts when calling the web service.
One of the biggest issues was the challenge of being able to correlate a message from BizTalk to the IIS log in the .net application and the custom logs in the application especially when there was a fairly large number of servers hosting the web services. However the key moment came when we were able to identify a specific call which had taken 40 seconds to execute on the server (yes a long time I know but that's a different story!). Anyway we were able to identify that this had timed out on the BizTalk side. Based on the normal 2 minute timeout we knew something unexpected was going on.
From here I decided to do some experimentation and I wanted to start outside of BizTalk because my hunch was this was not a BizTalk behaviour but something which was being highlighted by BizTalk because of our large load.
Server-side - Sample Web Service
To begin with I created a sample web service. Nothing special just a vanilla asmx web service hosted in IIS6 on Windows 2003 Standard Edition. The web service is just a hello world style web service as shown in the below picture. The only key feature is that the server side web method has a 30 second sleep in it and will trace out some information before and after the thread is set to sleep.
In the configuration for this web service there again is nothing special it's pretty much the most plain simple web service you could build.
Client-Side
To begin looking at what was happening with our example I created a number of different ways to consume the web service.
SoapHttpClientProtocol Example
I created a small application which would use a normal proxy generated to call the web service. It would iterate around a loop and make calls using the begin/end methods so I can do this asynchronously. I would do a loop of 20 calls with the ConnectionManager configuration
section supporting only 5 concurrent connections to the server.
<connectionManagement>
<remove address="*"/>
<add address = "*" maxconnection = "12" />
<add address = "http://<ServerName>" maxconnection = "5" />
</connectionManagement>
</system.net>
The below picture shows an example of the service calling code, key points are:
I have configured the timeout of 40 seconds for the proxy
I am using the asynchronous methods on the proxy to call the web service
The Test
I would run the client and execute 21 calls to the web service.
The Results
Below is the client side trace showing what's happening on the client.
In the below diagram is the web service side trace showing what's happening on the server
Some observations on the results are:
All of the calls were successful from the clients perspective
You could see the next call starting on the server as soon as the previous one had completed
Calls took significantly longer than 40 seconds from the start of our call to the return. In fact call 20 took 2 minutes and 30 seconds from the perspective of my code to execute even though I had set the timeout to 40 seconds
WSE 2 Sample
In the second example I used the exact same code to call the web service again with a single exception that I modified the web service proxy to derive from WebServiceClient protocol which is part of WSE 2 (using SP3). The below picture shows the basic code and the key points are:
I have configured the timeout of 40 seconds for the proxy
I am using the asynchronous methods on the proxy to call the web service
The Test
This test would execute 21 calls from the client to the web service.
The Results
The below trace is from the client side:
The below trace is from the server side:
Some observations on the trace results for this scenario are:
With call 4 if you look at the server side trace it did not start executing on the server for a number of seconds after the other 4 initial calls which were accepted by the server. I re-ran the test and this happened a couple of times and not on most others so at this point I'm just putting this down to something unexpected happening on the development machine and we will leave this observation out of scope of this article.
You can see that the client side trace statement executed almost immediately in all cases
All calls after the initial few calls would timeout
On the client side the calls that did timeout; timed out in a longer duration than the 40 seconds we set as the timeout
You can see that as calls were completing on the server the next calls were starting to come through
The calls that timed out on the client did actually connect to the server and their server side execution completed successfully
Elaboration on the findings
Based on the above observations I have drawn the below sequence diagram to illustrate conceptually what is happening. Everything except the final web service object is on the client side of the call. In the diagram below I've put two notes on the Web Service Proxy to show the two different places where the different base classes seem to start their timeout counters.
From the earlier samples we can work out that the timeout counter for the WSE web service proxy starts before the one for the SoapHttpClientProtocol proxy and the WSE one includes the time to get a connection from the pool; whereas the Soap proxy timeout just covers the method execution.
One interesting observation is if we rerun the above sample and increase the number of calls from 21 to 100,000 then for the WSE sample we will see a similar pattern where everything after the first few calls will timeout on the client as soon as it makes a connection to the server whereas the soap proxy will happily plug away and process all of the calls without a single timeout. I have actually set the sample running overnight and this did happen.
At this point you are probably thinking the same thoughts I was at the time about the differences in behaviour and which is right and why are they different? I'm not sure there is a definitive answer to this in the documentation, or at least not that I could find! I think you just have to consider that they are different and they could have different effects depending on your messaging solution.
In lots of situations this is just not an issue as your concurrent requests doesn't get to the situation where you end up throttling the web service calls on the client side, however this is definitely more common with an integration broker such as BizTalk where you often have high throughput requirements.
Some of the considerations you should make
Based on this behaviour you should be aware of the following:
In a .net application if you are making lots of concurrent web service calls from an application in an asynchronous manner your user may thing they are experiencing poor performance but you think your web service is working well. The problem could be that the client will have a default of 2 connections to remote servers so you should bear this in mind
When you are developing a BizTalk solution or a .net solution with the WSE 2 stack you may experience timeouts under load and throttling the number of connections using the max connections element in the configuration file will not help you
For an application using WSE2 or SoapHttpClientProtocol an expired timeout will not throw an error until after a connection to the server has been made so you should consider this in your transaction and durability patterns
Our Work Around
In the short term for our specific scenario we know that we can handle this by just increasing our timeout value. There is only a specific small window when we get lots of concurrent traffic that causes this scenario so we should be able to increase the timeout to take into consideration the additional client side wait, and on the odd occasion where we do get a timeout the BizTalk send port retry will handle this.
What was causing our original problem was that for that short window we were getting a lot of retries which significantly increased the load on our send servers and highlighted the issue.
Longer Term Solution
As a longer term solution this really gives us more ammunition to argue a migration to WCF. The application we are calling has some factors which limit the protocols we can use but with WCF we would have more control on the various timeout options because in WCF you can configure specific parts of the timeout.
Summary
I've had this blog post on my to do list for ages but hopefully it will be useful to some people to just understand this behaviour and to possibly help you with some performance issues you may have. I do not believe there is too much in the way of documentation particularly around WSE2 and ASMX in this area so again another bit of ammunition for migrating to WCF.
I'll try to do a follow up post with the sample for WCF to show how this changes things.
