Yikes, I ran into a real bummer of an edge case yesterday in one of my older low level handler implementations (for West Wind Web Connection in this case). Basically this handler is a connector for a backend Web framework that creates self contained HTTP output. An ASP.NET Handler captures the full output, and then shoves the result down the ASP.NET Response object pipeline writing out the content into the Response.OutputStream and seperately sending the HttpHeaders in the Response.Headers collection.

The headers turned out to be the problem and specifically Http Cookies, which for some reason ended up getting stripped out in some scenarios.

My handler works like this: Basically the HTTP response from the backend app would return a full set of HTTP headers plus the content. The ASP.NET handler would read the headers one at a time and then dump them out via Response.AppendHeader().

But I found that in some situations Set-Cookie headers sent along were simply stripped inside of the Http Handler. After a bunch of back and forth with some folks from Microsoft (thanks Damien and Levi!) I managed to pin this down to a very narrow edge scenario.

It's easiest to demonstrate the problem with a simple example HttpHandler implementation. The following simulates the very much simplified output generation process that fails in my handler. Specifically I have a couple of headers including a Set-Cookie header and some output that gets written into the Response object.

using System.Web;

namespace wwThreads
{

    public class Handler : IHttpHandler
    {
        /* NOTE: 
         * 
         * Run as a web.config set handler (see entry below)
         *
         * Best way is to look at the HTTP Headers in Fiddler
         * or Chrome/FireBug/IE tools and look for the 
         * WWHTREADSID cookie in the outgoing Response headers
         * ( If the cookie is not there you see the problem! )
         */
        public void ProcessRequest(HttpContext context)
        {                        
            HttpRequest request = context.Request;
            HttpResponse response = context.Response;

            // If ClearHeaders is used Set-Cookie header gets removed!
            // if commented header is sent...
            response.ClearHeaders();

            response.ClearContent(); 
                                
            // Demonstrate that other headers make it
            response.AppendHeader("RequestId", "asdasdasd");
            
            // This cookie gets removed when ClearHeaders above is called
            // When ClearHEaders is omitted above the cookie renders
            response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

           // *** This always works, even when explicit 
           //     Set-Cookie above fails and ClearHeaders is called
           //response.Cookies.Add(new HttpCookie("WWTHREADSID", "ThisIsTheValue"));

            response.Write(@"Output was created.<hr/>
                           Check output with Fiddler or HTTP Proxy to see whether cookie was sent.");            
        }

        public bool IsReusable
        {
            get { return false; }
        }

    }
}

In order to see the problem behavior this code has to be inside of an HttpHandler, and specifically in a handler defined in web.config with:

      <add name=".ck_handler"
           path="handler.ck"
           verb="*"
           type="wwThreads.Handler"
           preCondition="integratedMode" />

Note: Oddly enough this problem manifests only when configured through web.config, not in an ASHX handler, nor if you paste that same code into an ASPX page or MVC controller.

What's the problem exactly?

The code above simulates the more complex code in my live handler that picks up the HTTP response from the backend application and then peels out the headers and sends them one at a time via Response.AppendHeader. One of the headers in my app can be one or more Set-Cookie.

I found that the Set-Cookie headers were not making it into the Response headers output. Here's the Chrome Http Inspector trace:

Notice, no Set-Cookie header in the Response headers!

Now, running the very same request after removing the call to Response.ClearHeaders() command, the cookie header shows up just fine:

As you might expect it took a while to track this down. At first I thought my backend was not sending the headers but after closer checks I found that indeed the headers were set in the backend HTTP response, and they were indeed getting set via Response.AppendHeader() in the handler code. Yet, no cookie in the output.

In the simulated example the problem is this line:

response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

which in my live code is more dynamic ( ie. AppendHeader(token[0],token[1[]) )as it parses through the headers.

Bizzaro Land: Response.ClearHeaders() causes Cookie to get stripped

Now, here is where it really gets bizarre: The problem occurs only if:

• Response.ClearHeaders() was called before headers are added
• It only occurs in Http Handlers declared in web.config

Clearly this is an edge of an edge case but of course - knowing my relationship with Mr. Murphy - I ended up running smack into this problem.

So in the code above if you remove the call to ClearHeaders(), the cookie gets set!  Add it back in and the cookie is not there.

If I run the above code in an ASHX handler it works. If I paste the same code (with a Response.End()) into an ASPX page, or MVC controller it all works. Only in the HttpHandler configured through Web.config does it fail!

Cue the Twilight Zone Music.

Workarounds

As is often the case the fix for this once you know the problem is not too difficult. The difficulty lies in tracking inconsistencies like this down. Luckily there are a few simple workarounds for the Cookie issue.

Don't use AppendHeader for Cookies

The easiest and obvious solution to this problem is simply not use Response.AppendHeader() to set Cookies. Duh! Under normal circumstances in application level code there's rarely a reason to write out a cookie like this:

response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

but rather create the cookie using the Response.Cookies collection:

response.Cookies.Add(new HttpCookie("WWTHREADSID", "ThisIsTheValue"));

Unfortunately, in my case where I dynamically read headers from the original output and then dynamically  write header key value pairs back  programmatically into the Response.Headers collection, I actually don't look at each header specifically so in my case the cookie is just another header.

My first thought was to simply trap for the Set-Cookie header and then parse out the cookie and create a Cookie object instead. But given that cookies can have a lot of different options this is not exactly trivial, plus I don't really want to fuck around with cookie values which can be notoriously brittle.

Don't use Response.ClearHeaders()

The real mystery in all this is why calling Response.ClearHeaders() prevents a cookie value later written with Response.AppendHeader() to fail. I fired up Reflector and took a quick look at System.Web and HttpResponse.ClearHeaders. There's all sorts of resetting going on but nothing that seems to indicate that headers should be removed later on in the request. The code in ClearHeaders() does access the HttpWorkerRequest, which is the low level interface directly into IIS, and so I suspect it's actually IIS that's stripping the headers and not ASP.NET, but it's hard to know. Somebody from Microsoft and the IIS team would have to comment on that.

In my application it's probably safe to simply skip ClearHeaders() in my handler. The ClearHeaders/ClearContent was mainly for safety but after reviewing my code there really should never be a reason that headers would be set prior to this method firing.

However, if for whatever reason headers do need to be cleared, it's easy enough to manually clear the headers out:

private void RemoveHeaders(HttpResponse response)
{
    List<string> headers = new List<string>();
    foreach (string header in response.Headers)
    {
        headers.Add(header);
    }
    foreach (string header in headers)
    {
        response.Headers.Remove(header);
    }

    response.Cookies.Clear();
}

Now I can replace the call the Response.ClearHeaders() and I don't get the funky side-effects from Response.ClearHeaders().

Summary

I realize this is a total edge case as this occurs only in HttpHandlers that are manually configured. It looks like you'll never run into this in any of the higher level ASP.NET frameworks or even in ASHX handlers - only web.config defined handlers - which is really, really odd. After all those frameworks use the same underlying ASP.NET architecture. Hopefully somebody from Microsoft has an idea what crazy dependency was triggered here to make this fail.

IAC, there are workarounds to this should you run into it, although I bet when you do run into it, it'll likely take a bit of time to find the problem or even this post in a search because it's not easily to correlate the problem to the solution. It's quite possible that more than cookies are affected by this behavior. Searching for a solution I read a few other accounts where headers like Referer were mysteriously disappearing, and it's possible that something similar is happening in those cases.

Again, extreme edge case, but I'm writing this up here as documentation for myself and possibly some others that might have run into this.