During one of my Handlers and Modules session at DevConnections this week one of
the attendees asked a question that I didn’t have an immediate answer for. Basically he wanted
to capture response output completely and then apply some filtering
to the output – effectively injecting some additional content into
the page AFTER
the page had completely rendered. Specifically
the output should be captured from anywhere – not just a page and have this code injected into
the page. Some time ago I posted some code that allows you
to capture ASP.NET Page output by overriding
the Render() method, capturing
the HtmlTextWriter() and reading its content, modifying
the rendered data as text then writing it
back out. I’ve actually used this approach on a few occasions and it works fine for ASP.NET pages. But this obviously won’t work outside of
the Page class environment and it’s not really generic – you have
to create a custom page class in order
to handle
the output capture. [updated 11/16/2009 – updated ResponseFilterStream implementation and a few additional notes based on comments] Enter Response.Filter However, ASP.NET includes a Response.Filter which can be used – well
to filter output. Basically Response.Filter is a stream through which
the OutputStream is piped
back to the Web Server (indirectly). As content is written into
the Response object,
the filter stream receives
the appropriate Stream commands like Write, Flush and Close as well as read operations although for a Response.Filter that’s uncommon
to be hit.
The Response.Filter can be programmatically replaced at runtime which allows you
to effectively intercept all output generation that runs through ASP.NET. A common Example: Dynamic GZip Encoding A rather common use of Response.Filter hooking up code based, dynamic GZip compression for requests which is dead simple by applying a GZipStream (or DeflateStream)
to Response.Filter.
The following generic routines can be used very easily
to detect GZip capability of
the client and compress response output with a single line of code and a couple of library helper routines: WebUtils.GZipEncodePage();
which is handled with a few lines of reusable code and a couple of static helper methods:
/// <summary>
///Sets up
the current page or handler
to use GZip through a Response.Filter
///IMPORTANT: ///You have
to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
HttpResponse Response = HttpContext.Current.Response;
if(IsGZipSupported())
{
stringAcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
if(AcceptEncoding.Contains("deflate"))
{
Response.Filter = newSystem.IO.Compression.DeflateStream(Response.Filter,
System.IO.Compression.CompressionMode.Compress);
Response.AppendHeader("Content-Encoding", "deflate");
}
else
{
Response.Filter = newSystem.IO.Compression.GZipStream(Response.Filter,
System.IO.Compression.CompressionMode.Compress);
Response.AppendHeader("Content-Encoding", "gzip"); }
}
// Allow proxy servers
to cache encoded and unencoded versions separately
Response.AppendHeader("Vary", "Content-Encoding");
}
/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
if (!string.IsNullOrEmpty(AcceptEncoding) &&
(AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))
return true;
return false;
}
GZipStream and DeflateStream are streams that are assigned
to Response.Filter and by doing so apply
the appropriate compression on
the active Response.
Response.Filter content is chunked
So
to implement a Response.Filter effectively requires only that you implement a custom stream and handle
the Write() method
to capture Response output as it’s written. At first blush this seems very simple – you capture
the output in Write, transform it and write out
the transformed content in one pass. And that indeed works for small amounts of content. But you see,
the problem is that output is written in small buffer chunks (a little less than 16k it appears) rather than just a single Write() statement into
the stream, which makes perfect sense for ASP.NET
to stream data
back to IIS in smaller chunks
to minimize memory usage en route.
Unfortunately this also makes it a more difficult
to implement any filtering routines since you don’t directly get access
to all of
the response content which is problematic especially if those filtering routines require you
to look at
the ENTIRE response in order
to transform or capture
the output as is needed for
the solution
the gentleman in my session asked for.
So in order
to address this a slightly different approach is required that basically captures all
the Write() buffers passed into a cached stream and then making
the stream available only when it’s complete and ready
to be flushed.
As I was thinking about
the implementation I also started thinking about
the few instances when I’ve used Response.Filter implementations. Each time I had
to create a new Stream subclass and create my custom functionality but in
the end each implementation did
the same thing – capturing output and transforming it. I thought there should be an easier way
to do this by creating a re-usable Stream class that can handle stream transformations that are common
to Response.Filter implementations.
Creating a semi-generic Response Filter Stream Class
What I ended up with is a ResponseFilterStream class that provides a handful of Events that allow you
to capture and/or transform Response content.
The class implements a subclass of Stream and then overrides Write() and Flush()
to handle capturing and transformation operations. By exposing events it’s easy
to hook up capture or transformation operations via single focused methods.
ResponseFilterStream exposes
the following events:
CaptureStream, CaptureString
Captures
the output only and provides either a MemoryStream or String with
the final page output. Capture is hooked
to the Flush() operation of
the stream.
TransformStream, TransformString
Allows you
to transform
the complete response output with events that receive a MemoryStream or String respectively and can you modify
the output then return it
back as a return value.
The transformed output is then written
back out in a single chunk
to the response output stream. These events capture all output internally first then write
the entire buffer into
the response.
TransformWrite, TransformWriteString
Allows you
to transform
the Response data as it is written in its original chunk size in
the Stream’s Write() method. Unlike TransformStream/TransformString which operate on
the complete output, these events only see
the current chunk of data written. This is more efficient as there’s no caching involved, but can cause problems due
to searched content splitting over multiple chunks.
Using this implementation, creating a custom Response.Filter transformation becomes as simple as
the following code.
To hook up
the Response.Filter using
the MemoryStream version event:
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformStream += filter_TransformStream;
Response.Filter = filter;
and
the event handler
to do
the transformation:
MemoryStream filter_TransformStream(MemoryStream ms)
{
Encoding encoding = HttpContext.Current.Response.ContentEncoding;
string output = encoding.GetString(ms.ToArray());
output = FixPaths(output);
ms = new MemoryStream(output.Length);
byte[] buffer = encoding.GetBytes(output);
ms.Write(buffer,0,buffer.Length);
return ms;
}
private string FixPaths(string output)
{
string path = HttpContext.Current.Request.ApplicationPath; // override root path wonkiness
if (path == "/")
path = "";
output = output.Replace("\"~/", "\"" + path + "/").Replace("'~/", "'" + path + "/");
return output;
}
The idea of
the event handler is that you can do whatever you want
to the stream and return
back a stream – either
the same one that’s been modified or a brand new one – which is then sent
back to as
the final response.
The above code can be simplified even more by using
the string version events which handle
the stream
to string conversions for you:
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformString += filter_TransformString;
Response.Filter = filter;
and
the event handler
to do
the transformation calling
the same FixPaths method shown above:
string filter_TransformString(string output)
{
return FixPaths(output);
}
The events for capturing output and capturing and transforming chunks work in a very similar way. By using events
to handle
the transformations ResponseFilterStream becomes a reusable component and we don’t have
to create a new stream class or subclass an existing Stream based classed.
By
the way,
the example used here is kind of a cool trick which transforms “~/” expressions inside of
the final generated HTML output – even in plain HTML controls not HTML controls – and transforms them into
the appropriate application relative path in
the same way that ResolveUrl would do.
So you can write plain old HTML like this:
<a href=”~/default.aspx”>Home</a>
and have it turned into:
<a href=”/myVirtual/default.aspx”>Home</a>
without having
to use an ASP.NET control like Hyperlink or Image or having
to constantly use:
<img src=”<%= ResolveUrl(“~/images/home.gif”) %>” />
in MVC applications (which frankly is one of
the most annoying things about MVC especially given
the path hell that extension-less and endpoint-less URLs impose).
I can’t take credit for this idea. While discussing
the Response.Filter issues on Twitter a hint from Dylan Beattie who pointed me at one of his examples which does something similar. I thought
the idea was cool enough
to use an example for
future demos of Response.Filter functionality in ASP.NET next I time I do
the Modules and Handlers talk (which was great fun BTW).
How practical this is is debatable however since there’s definitely some overhead
to using a Response.Filter in general and especially on one that caches
the output and
the re-writes it later. Make sure
to test for performance anytime you use Response.Filter hookup and make sure it' doesn’t end up killing perf on you. You’ve been warned :-}.
How does ResponseFilterStream work?
The big win of this implementation IMHO is that it’s a reusable component – so for implementation there’s no new class, no subclassing – you simply attach
to an event
to implement an event handler method with a straight forward signature
to retrieve
the stream or string you’re interested in.
The implementation is based on a subclass of Stream as is required in order
to handle
the Response.Filter requirements. What’s different than other implementations I’ve seen in various places is that it supports capturing output as a whole
to allow retrieving
the full response output for capture or modification.
The exception are
the TransformWrite and TransformWrite events which operate only active chunk of data written by
the Response.
For captured output,
the Write() method captures output into an internal MemoryStream that is cached until writing is complete. So Write() is called when ASP.NET writes
to the Response stream, but
the filter doesn’t pass on
the Write immediately
to the filter’s internal stream.
The data is cached and only when
the Flush() method is called
to finalize
the Stream’s output do we actually send
the cached stream off for transformation (if
the events are hooked up) and THEN finally write out
the returned content in one big chunk.
Here’s
the implementation of ResponseFilterStream:
/// <summary>
/// A semi-generic Stream implementation for Response.Filter with
/// an event interface for handling Content transformations via
/// Stream or String.
/// <remarks>
/// Use with care for large output as this implementation copies
///
the output into a memory stream and so increases memory usage.
/// </remarks>
/// </summary>
public class ResponseFilterStream : Stream
{
/// <summary>
///
The original stream
/// </summary>
Stream _stream;
/// <summary>
/// Current position in
the original stream
/// </summary>
long _position;
/// <summary>
/// Stream that original content is read into
/// and then passed
to TransformStream function
/// </summary>
MemoryStream _cacheStream = new MemoryStream(5000);
/// <summary>
/// Internal pointer that that keeps track of
the size
/// of
the cacheStream
/// </summary>
int _cachePointer = 0;
/// <summary>
///
/// </summary>
/// <param name="responseStream"></param>
public ResponseFilterStream(Stream responseStream)
{
_stream = responseStream;
}
/// <summary>
/// Determines whether
the stream is captured
/// </summary>
private bool IsCaptured
{
get
{
if (CaptureStream != null || CaptureString != null ||
TransformStream != null || TransformString != null)
return true;
return false;
}
}
/// <summary>
/// Determines whether
the Write method is outputting data immediately
/// or delaying output until Flush() is fired.
/// </summary>
private bool IsOutputDelayed
{
get
{
if (TransformStream != null || TransformString != null)
return true;
return false;
}
}
/// <summary>
/// Event that captures Response output and makes it available
/// as a MemoryStream instance. Output is captured but won't
/// affect Response output.
/// </summary>
public event Action<MemoryStream> CaptureStream;
/// <summary>
/// Event that captures Response output and makes it available
/// as a string. Output is captured but won't affect Response output.
/// </summary>
public event Action<string> CaptureString;
/// <summary>
/// Event that allows you transform
the stream as each chunk of
///
the output is written in
the Write() operation of
the stream.
/// This means that that it's possible/likely that
the input
/// buffer will not contain
the full response output but only
/// one of potentially many chunks.
///
/// This event is called as part of
the filter stream's Write()
/// operation.
/// </summary>
public event Func<byte[], byte[]> TransformWrite;
/// <summary>
/// Event that allows you
to transform
the response stream as
/// each chunk of bytep[] output is written during
the stream's write
/// operation. This means it's possibly/likely that
the string
/// passed
to the handler only contains a portion of
the full
/// output. Typical buffer chunks are around 16k a piece.
///
/// This event is called as part of
the stream's Write operation.
/// </summary>
public event Func<string, string> TransformWriteString;
/// <summary>
/// This event allows capturing and transformation of
the entire
/// output stream by caching all write operations and delaying final
/// response output until Flush() is called on
the stream.
/// </summary>
public event Func<MemoryStream, MemoryStream> TransformStream;
/// <summary>
/// Event that can be hooked up
to handle Response.Filter
/// Transformation. Passed a string that you can modify and
/// return
back as a return value.
The modified content
/// will become
the final output.
/// </summary>
public event Func<string, string> TransformString;
protected virtual void OnCaptureStream(MemoryStream ms)
{
if (CaptureStream != null)
CaptureStream(ms);
}
private void OnCaptureStringInternal(MemoryStream ms)
{
if (CaptureString != null)
{
string content = HttpContext.Current.Response.ContentEncoding.GetString(ms.ToArray());
OnCaptureString(content);
}
}
protected virtual void OnCaptureString(string output)
{
if (CaptureString != null)
CaptureString(output);
}
protected virtual byte[] OnTransformWrite(byte[] buffer)
{
if (TransformWrite != null)
return TransformWrite(buffer);
return buffer;
}
private byte[] OnTransformWriteStringInternal(byte[] buffer)
{
Encoding encoding = HttpContext.Current.Response.ContentEncoding;
string output = OnTransformWriteString(encoding.GetString(buffer));
return encoding.GetBytes(output);
}
private string OnTransformWriteString(string value)
{
if (TransformWriteString != null)
return TransformWriteString(value);
return value;
}
protected virtual MemoryStream OnTransformCompleteStream(MemoryStream ms)
{
if (TransformStream != null)
return TransformStream(ms);
return ms;
}
/// <summary>
/// Allows transforming of strings
///
/// Note this handler is internal and not meant
to be overridden
/// as
the TransformString Event has
to be hooked up in order
/// for this handler
to even fire
to avoid
the overhead of string
/// conversion on every pass through.
/// </summary>
/// <param name="responseText"></param>
/// <returns></returns>
private string OnTransformCompleteString(string responseText)
{
if (TransformString != null)
TransformString(responseText);
return responseText;
}
/// <summary>
/// Wrapper method form OnTransformString that handles
/// stream
to string and vice versa conversions
/// </summary>
/// <param name="ms"></param>
/// <returns></returns>
internal MemoryStream OnTransformCompleteStringInternal(MemoryStream ms)
{
if (TransformString == null)
return ms;
//string content = ms.GetAsString();
string content = HttpContext.Current.Response.ContentEncoding.GetString(ms.ToArray());
content = TransformString(content);
byte[] buffer = HttpContext.Current.Response.ContentEncoding.GetBytes(content);
ms = new MemoryStream();
ms.Write(buffer, 0, buffer.Length);
//ms.WriteString(content);
return ms;
}
/// <summary>
///
/// </summary>
public override bool CanRead
{
get { return true; }
}
public override bool CanSeek
{
get { return true; }
}
/// <summary>
///
/// </summary>
public override bool CanWrite
{
get { return true; }
}
/// <summary>
///
/// </summary>
public override long Length
{
get { return 0; }
}
/// <summary>
///
/// </summary>
public override long Position
{
get { return _position; }
set { _position = value; }
}
/// <summary>
///
/// </summary>
/// <param name="offset"></param>
/// <param name="direction"></param>
/// <returns></returns>
public override long Seek(long offset, System.IO.SeekOrigin direction)
{
return _stream.Seek(offset, direction);
}
/// <summary>
///
/// </summary>
/// <param name="length"></param>
public override void SetLength(long length)
{
_stream.SetLength(length);
}
/// <summary>
///
/// </summary>
public override void Close()
{
_stream.Close();
}
/// <summary>
/// Override flush by writing out
the cached stream data
/// </summary>
public override void Flush()
{
if (IsCaptured && _cacheStream.Length > 0)
{
// Check for transform implementations
_cacheStream = OnTransformCompleteStream(_cacheStream);
_cacheStream = OnTransformCompleteStringInternal(_cacheStream);
OnCaptureStream(_cacheStream);
OnCaptureStringInternal(_cacheStream);
// write
the stream
back out if output was delayed
if (IsOutputDelayed)
_stream.Write(_cacheStream.ToArray(), 0, (int)_cacheStream.Length);
// Clear
the cache once we've written it out
_cacheStream.SetLength(0);
}
// default flush behavior
_stream.Flush();
}
/// <summary>
///
/// </summary>
/// <param name="buffer"></param>
/// <param name="offset"></param>
/// <param name="count"></param>
/// <returns></returns>
public override int Read(byte[] buffer, int offset, int count)
{
return _stream.Read(buffer, offset, count);
}
/// <summary>
/// Overriden
to capture output written by ASP.NET and captured
/// into a cached stream that is written out later when Flush()
/// is called.
/// </summary>
/// <param name="buffer"></param>
/// <param name="offset"></param>
/// <param name="count"></param>
public override void Write(byte[] buffer, int offset, int count)
{
if ( IsCaptured )
{
// copy
to holding buffer only - we'll write out later
_cacheStream.Write(buffer, 0, count);
_cachePointer += count;
}
// just transform this buffer
if (TransformWrite != null)
buffer = OnTransformWrite(buffer);
if (TransformWriteString != null)
buffer = OnTransformWriteStringInternal(buffer);
if (!IsOutputDelayed)
_stream.Write(buffer, offset, buffer.Length);
}
}
The key features are
the events and corresponding OnXXX methods that handle
the event hookups, and
the Write() and Flush() methods of
the stream implementation. All
the rest of
the members tend
to be plain jane passthrough stream implementation code without much consequence.
I do love
the way Action<t> and Func<T> make it so easy
to create
the event signatures for
the various events – sweet.
A few Things
to consider
Performance
Response.Filter is not great for performance in general as it adds another layer of indirection
to the ASP.NET output pipeline, and this implementation in particular adds a memory hit as it basically duplicates
the response output into
the cached memory stream which is necessary since you may have
to look at
the entire response. If you have large pages in particular this can cause potentially serious memory pressure in your server application. So be careful of wholesale adoption of this (or other) Response.Filters. Make sure
to do some performance testing
to ensure it’s not killing your app’s performance.
Response.Filter works everywhere
A few questions came up in comments and discussion as
to capturing ALL output hitting
the site and – yes you can definitely do that by assigning a Response.Filter inside of a module. If you do this however you’ll want
to be very careful and decide which content you actually want
to capture especially in IIS 7 which passes ALL content – including static images/CSS etc. through
the ASP.NET pipeline. So it is important
to filter only on what you’re looking for – like
the page extension or maybe more effectively
the Response.ContentType.
Response.Filter Chaining
Originally I thought that filter chaining doesn’t work at all due
to a bug in
the stream implementation code. But it’s quite possible
to assign multiple filters
to the Response.Filter property. So
the following actually works
to both compress
the output and apply
the transformed content:
WebUtils.GZipEncodePage();
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformString += filter_TransformString;
Response.Filter = filter;
However
the following does not work resulting in invalid content encoding errors:
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformString += filter_TransformString;
Response.Filter = filter;
WebUtils.GZipEncodePage();
In other words multiple Response filters can work together but it depends entirely on
the implementation whether they can be chained or in which order they can be chained. In this case running
the GZip/Deflate stream filters apparently relies on
the original content length of
the output and chokes when
the content is modified. But if attaching
the compression first it works fine as unintuitive as that may seem.
Resources
Download example code
Capture Output from ASP.NET Pages
© Rick Strahl, West Wind Technologies, 2005-2010Posted in ASP.NET
