Search engine optimization (SEO) is important for any publically facing web-site. A large % of traffic to sites now comes directly from search engines, and improving your site’s search relevancy will lead to more users visiting your site from search engine queries. This can directly or indirectly increase
the money you make through your site. This blog post covers how you can use
the free Microsoft URL Rewrite Extension to fix a bunch of common SEO problems that your site might have. It takes less than 15 minutes (and no code changes) to apply 4 simple URL Rewrite rules to your site, and in doing so cause search engines to drive more visitors and traffic to your site.
The techniques below work equally well with both ASP.NET Web Forms and ASP.NET MVC based sites. They also works with all versions of ASP.NET (and even work with non-ASP.NET content). [In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu] Measuring
the SEO of your website with
the Microsoft SEO Toolkit A few months ago I blogged about
the free SEO Toolkit that we’ve shipped. This useful tool enables you to automatically crawl/scan your site for SEO correctness, and it then flags any SEO issues it finds. I highly recommend downloading and using
the tool against any public site you work on. It makes it easy to spot SEO issues you might have in your site, and pinpoint ways to optimize it further. Below is a simple example of a report I ran against one of my sites (www.scottgu.com) prior to applying
the URL Rewrite rules I’ll cover later in this blog post: Search Relevancy and URL Splitting Two of
the important things that search engines evaluate when assessing your site’s “search relevancy” are: How many other sites link to your content. Search engines assume that if a lot of people around
the web are linking to your content, then it is likely useful and so weight it higher in relevancy.
The uniqueness of
the content it finds on your site. If search engines find that
the content is duplicated in multiple places around
the Internet (or on multiple URLs on your site) then it is likely to drop
the relevancy of
the content. One of
the things you want to be very careful to avoid when building public facing sites is to not allow different URLs to retrieve
the same content within your site. Doing so will hurt with both of
the situations above. In particular, allowing external sites to link to
the same content with multiple URLs will cause your link-count and page-ranking to be split up across those different URLs (and so give you a smaller page rank than what it would otherwise be if it was just one URL). Not allowing external sites to link to you in different ways sounds easy in theory – but you might wonder what exactly this means in practice and how you avoid it. 4 Really Common SEO Problems Your Sites Might Have Below are 4 really common scenarios that can cause your site to inadvertently expose multiple URLs for
the same content. When this happens external sites linking to yours will end up splitting their page links across multiple URLs - and as a result cause you to have a lower page ranking with search engines than you deserve. SEO Problem #1: Default Document IIS (and other web servers) supports
the concept of a “default document”. This allows you to avoid having to explicitly specify
the page you want to serve at either
the root of
the web-site/application, or within a sub-directory. This is convenient – but means that by default this content is available via two different publically exposed URLs (which is bad). For example: http://scottgu.com/ http://scottgu.com/default.aspx SEO Problem #2: Different URL Casings Web developers often don’t realize URLs are case sensitive to search engines on
the web. This means that search engines will treat
the following links as two completely different URLs: http://scottgu.com/Albums.aspx http://scottgu.com/albums.aspx SEO Problem #3: Trailing Slashes Consider
the below two URLs – they might look
the same at first, but they are subtly different.
The trailing slash creates yet another situation that causes search engines to treat
the URLs as different and so split search rankings: http://scottgu.com http://scottgu.com/ SEO Problem #4: Canonical Host Names Sometimes sites support scenarios where they support a web-site with both a leading “www” hostname prefix as well as just
the hostname itself. This causes search engines to treat
the URLs as different and split search rankling: http://scottgu.com/albums.aspx/ http://www.scottgu.com/albums.aspx/ How to Easily Fix these SEO Problems in 10 minutes (or less) using IIS Rewrite If you haven’t been careful when coding your sites, chances are you are suffering from one (or more) of
the above SEO problems. Addressing these issues will improve your search engine relevancy ranking and drive more traffic to your site.
The “good news” is that fixing
the above 4 issues is really easy using
the URL Rewrite Extension. This is a completely free Microsoft extension available for IIS 7.x (on Windows Server 2008, Windows Server 2008 R2, Windows 7 and Windows Vista).
The great thing about using
the IIS Rewrite extension is that it allows you to fix
the above problems *without* having to change any code within your applications. You can easily install
the URL Rewrite Extension in under 3 minutes using
the Microsoft Web Platform Installer (a free tool we ship that automates setting up web servers and development machines). Just click
the green “Install Now” button on
the URL Rewrite Spotlight page to install it on your Windows Server 2008, Windows 7 or Windows Vista machine: Once installed you’ll find that a new “URL Rewrite” icon is available within
the IIS 7 Admin Tool: Double-clicking
the icon will open up
the URL Rewrite admin panel – which will display
the list of URL Rewrite rules configured for a particular application or site: Notice that our rewrite rule list above is currently empty (which is
the default when you first install
the extension). We can click
the “Add Rule…” link button in
the top-right of
the panel to add and enable new URL Rewriting logic for our site. Scenario 1: Handling Default Document Scenarios One of
the SEO problems I discussed earlier in this post was
the scenario where
the “default document” feature of IIS causes you to inadvertently expose two URLs for
the same content on your site. For example: http://scottgu.com/ http://scottgu.com/default.aspx We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates to
the second URL to instead go to
the first one. We will setup
the HTTP redirect to be a “permanent redirect” – which will indicate to search engines that they should follow
the redirect and use
the new URL they are redirected to as
the identifier of
the content they retrieve. Let’s look at how we can create such a rule. We’ll begin by clicking
the “Add Rule” link in
the screenshot above. This will cause
the below dialog to display: We’ll select
the “Blank Rule” template within
the “Inbound rules” section to create a new custom URL Rewriting rule. This will display an empty pane like below: Don’t worry – setting up
the above rule is easy.
The following 4 steps explain how to do so: Step 1: Name
the Rule Our first step will be to name
the rule we are creating. Naming it with a descriptive name will make it easier to find and understand later. Let’s name this rule our “Default Document URL Rewrite” rule: Step 2: Setup
the Regular Expression that Matches this Rule Our second step will be to specify a regular expression filter that will cause this rule to execute when an incoming URL matches
the regex pattern. Don’t worry if you aren’t good with regular expressions - I suck at them too.
The trick is to know someone who is good at them or copy/paste them from a web-site. Below we are going to specify
the following regular expression as our pattern rule: (.*?)/?Default\.aspx$ This pattern will match any URL string that ends with Default.aspx.
The "(.*?)" matches any preceding character zero or more times.
The "/?" part says to match
the slash symbol zero or one times.
The "$" symbol at
the end will ensure that
the pattern will only match strings that end with Default.aspx. Combining all these regex elements allows this rule to work not only for
the root of your web site (e.g. http://scottgu.com/default.aspx) but also for any application or subdirectory within
the site (e.g. http://scottgu.com/photos/default.aspx. Because
the “ignore case” checkbox is selected it will match both “Default.aspx” as well as “default.aspx” within
the URL. One nice feature built-into
the rule editor is a “Test pattern” button that you can click to bring up a dialog that allows you to test out a few URLs with
the rule you are configuring: Above I've added a “products/default.aspx” URL and clicked
the “Test” button. This will give me immediate feedback on whether
the rule will execute for it. Step 3: Setup a Permanent Redirect Action We’ll then setup an action to occur when our regular expression pattern matches
the incoming URL: In
the dialog above I’ve changed
the “Action Type” drop down to be a “Redirect” action.
The “Redirect Type” will be a HTTP 301 Permanent redirect – which means search engines will follow it. I’ve also set
the “Redirect URL” property to be: {R:1}/ This indicates that we want to redirect
the web client requesting
the original URL to a new URL that has
the originally requested URL path - minus
the "Default.aspx" in it. For example, requests for http://scottgu.com/default.aspx will be redirected to http://scottgu.com/, and requests for http://scottgu.com/photos/default.aspx will be redirected to http://scottgu.com/photos/
The "{R:N}" regex construct, where N >= 0, is called a back-reference and N is
the back-reference index. In
the case of our pattern "(.*?)/?Default\.aspx$", if
the input URL is "products/Default.aspx" then {R:0} will contain "products/Default.aspx" and {R:1} will contain "products". We are going to use this {R:1}/ value to be
the URL we redirect users to. Step 4: Apply and
Save the Rule Our final step is to click
the “Apply” button in
the top right hand of
the IIS admin tool – which will cause
the tool to persist
the URL Rewrite rule into our application’s root web.config file (under a <system.webServer/rewrite> configuration section): <configuration> <system.webServer> <rewrite> <rules> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Because IIS 7.x and ASP.NET share
the same web.config files, you can actually just copy/paste
the above code into your web.config files using Visual Studio and skip
the need to run
the admin tool entirely. This also makes adding/deploying URL Rewrite rules with your ASP.NET applications really easy. Step 5: Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://scottgu.com/ http://scottgu.com/default.aspx Notice that
the second URL automatically redirects to
the first one. Because it is a permanent redirect, search engines will follow
the URL and should update
the page ranking of http://scottgu.com to include links to http://scottgu.com/default.aspx as well. Scenario 2: Different URL Casing Another common SEO problem I discussed earlier in this post is that URLs are case sensitive to search engines on
the web. This means that search engines will treat
the following links as two completely different URLs: http://scottgu.com/Albums.aspx http://scottgu.com/albums.aspx We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates to
the first URL to instead go to
the second (all lower-case) one. Like before, we will setup
the HTTP redirect to be a “permanent redirect” – which will indicate to search engines that they should follow
the redirect and use
the new URL they are redirected to as
the identifier of
the content they retrieve. To create such a rule we’ll click
the “Add Rule” link in
the URL Rewrite admin tool again. This will cause
the “Add Rule” dialog to appear again: Unlike
the previous scenario (where we created a “Blank Rule”), with this scenario we can take advantage of a built-in “Enforce lowercase URLs” rule template. When we click
the “ok” button we’ll see
the following dialog which asks us if we want to create a rule that enforces
the use of lowercase letters in URLs: When we click
the “Yes” button we’ll get a pre-written rule that automatically performs a permanent redirect if an incoming URL has upper-case characters in it – and automatically send users to a lower-case version of
the URL: We can click
the “Apply” button to use this rule “as-is” and have it apply to all incoming URLs to our site. Because my www.scottgu.com site uses ASP.NET Web Forms, I’m going to make one small change to
the rule we generated above – which is to add a condition that will ensure that URLs to ASP.NET’s built-in “WebResource.axd” handler are excluded from our case-sensitivity URL Rewrite logic. URLs to
the WebResource.axd handler will only come from server-controls emitted from my pages – and will never be linked to from external sites. While my site will continue to function fine if we redirect these URLs to automatically be lower-case – doing so isn’t necessary and will add an extra HTTP redirect to many of my pages.
The good news is that adding a condition that prevents my URL Rewriting rule from happening with certain URLs is easy. We simply need to expand
the “Conditions” section of
the form above We can then click
the “Add” button to add a condition clause. This will bring up
the “Add Condition” dialog: Above I’ve entered {URL} as
the Condition input – and said that this rule should only execute if
the URL does not match a regex pattern which contains
the string “WebResource.axd”. This will ensure that WebResource.axd URLs to my site will be allowed to execute just fine without having
the URL be re-written to be all lower-case. Note: If you have static resources (like references to .jpg, .css, and .js files) within your site that currently use upper-case characters you’ll probably want to add additional condition filter clauses so that URLs to them also don’t get redirected to be lower-case (just add rules for patterns like .jpg, .gif, .js, etc). Your site will continue to work fine if these URLs get redirected to be lower case (meaning
the site won’t break) – but it will cause an extra HTTP redirect to happen on your site for URLs that don’t need to be redirected for SEO reasons. So setting up a condition clause makes sense to add. When I click
the “ok” button above and apply our lower-case rewriting rule
the admin tool will
save the following additional rule to our web.config file: <configuration> <system.webServer> <rewrite> <rules> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> <rule name="Lower Case URLs" stopProcessing="true"> <match url="[A-Z]" ignoreCase="false" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{ToLower:{URL}}" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://scottgu.com/Albums.aspx http://scottgu.com/albums.aspx Notice that
the first URL (which has a capital “A”) automatically does a redirect to a lower-case version of
the URL. Scenario 3: Trailing Slashes Another common SEO problem I discussed earlier in this post is
the scenario of trailing slashes within URLs.
The trailing slash creates yet another situation that causes search engines to treat
the URLs as different and so split search rankings: http://scottgu.com http://scottgu.com/ We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates to
the first URL (that does not have a trailing slash) to instead go to
the second one that does. Like before, we will setup
the HTTP redirect to be a “permanent redirect” – which will indicate to search engines that they should follow
the redirect and use
the new URL they are redirected to as
the identifier of
the content they retrieve. To create such a rule we’ll click
the “Add Rule” link in
the URL Rewrite admin tool again. This will cause
the “Add Rule” dialog to appear again:
The URL Rewrite admin tool has a built-in “Append or remove
the trailing slash symbol” rule template. When we select it and click
the “ok” button we’ll see
the following dialog which asks us if we want to create a rule that automatically redirects users to a URL with a trailing slash if one isn’t present: Like within our previous lower-casing rewrite rule we’ll add one additional condition clause that will exclude WebResource.axd URLs from being processed by this rule. This will avoid an unnecessary redirect for happening for those URLs. When we click
the “OK” button we’ll get a pre-written rule that automatically performs a permanent redirect if
the URL doesn’t have a trailing slash – and if
the URL is not processed by either a directory or a file. This will
save the following additional rule to our web.config file: <configuration> <system.webServer> <rewrite> <rules> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> <rule name="Lower Case URLs" stopProcessing="true"> <match url="[A-Z]" ignoreCase="false" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{ToLower:{URL}}" /> </rule> <rule name="Trailing Slash" stopProcessing="true"> <match url="(.*[^/])$" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" /> <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" /> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{R:1}/" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://scottgu.com http://scottgu.com/ Notice that
the first URL (which has no trailing slash) automatically does a redirect to a URL with
the trailing slash. Because it is a permanent redirect, search engines will follow
the URL and update
the page ranking. Scenario 4: Canonical Host Names
The final SEO problem I discussed earlier are scenarios where a site works with both a leading “www” hostname prefix as well as just
the hostname itself. This causes search engines to treat
the URLs as different and split search rankling: http://www.scottgu.com/albums.aspx http://scottgu.com/albums.aspx We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates to
the first URL (that has a www prefix) to instead go to
the second URL. Like before, we will setup
the HTTP redirect to be a “permanent redirect” – which will indicate to search engines that they should follow
the redirect and use
the new URL they are redirected to as
the identifier of
the content they retrieve. To create such a rule we’ll click
the “Add Rule” link in
the URL Rewrite admin tool again. This will cause
the “Add Rule” dialog to appear again:
The URL Rewrite admin tool has a built-in “Canonical domain name” rule template. When we select it and click
the “ok” button we’ll see
the following dialog which asks us if we want to create a redirect rule that automatically redirects users to a primary host name URL: Above I’m entering
the primary URL address I want to expose to
the web: scottgu.com. When we click
the “OK” button we’ll get a pre-written rule that automatically performs a permanent redirect if
the URL has another leading domain name prefix. This will
save the following additional rule to our web.config file: <configuration> <system.webServer> <rewrite> <rules> <rule name="Cannonical Hostname"> <match url="(.*)" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{HTTP_HOST}" pattern="^scottgu\.com$" negate="true" /> </conditions> <action type="Redirect" url="http://scottgu.com/{R:1}" /> </rule> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> <rule name="Lower Case URLs" stopProcessing="true"> <match url="[A-Z]" ignoreCase="false" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{ToLower:{URL}}" /> </rule> <rule name="Trailing Slash" stopProcessing="true"> <match url="(.*[^/])$" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" /> <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" /> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{R:1}/" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://www.scottgu.com/albums.aspx http://scottgu.com/albums.aspx Notice that
the first URL (which has
the “www” prefix) now automatically does a redirect to
the second URL which does not have
the www prefix. Because it is a permanent redirect, search engines will follow
the URL and update
the page ranking. 4 Simple Rules for Improved SEO
The above 4 rules are pretty easy to setup and should take less than 15 minutes to configure on existing sites you already have.
The beauty of using a solution like
the URL Rewrite Extension is that you can take advantage of it without having to change code within your web-site – and without having to break any existing links already pointing at your site. Users who follow existing links will be automatically redirected to
the new URLs you wish to publish. And search engines will start to give your site a higher search relevancy ranking – which will list your site higher in search results and drive more traffic to it. Customizing your URL Rewriting rules further is easy to-do either by editing
the web.config file directly, or alternatively, just double click
the URL Rewrite icon within
the IIS 7.x admin tool and it will list all
the active rules for your web-site or application: Clicking any of
the rules above will open
the rules editor back up and allow you to tweak/customize/save them further. Summary Measuring and improving SEO is something every developer building a public-facing web-site needs to think about and focus on. If you haven’t already, download and use
the SEO Toolkit to analyze
the SEO of your sites today. New URL Routing features in ASP.NET MVC and ASP.NET Web Forms 4 make it much easier to build applications that have more control over
the URLs that are published. Tools like
the URL Rewrite Extension that I’ve talked about in this blog post make it much easier to improve
the URLs that are published from sites you already have built today – without requiring you to change a lot of code.
The URL Rewrite Extension provides a bunch of additional great capabilities – far beyond just SEO - as well. I’ll be covering these additional capabilities more in future blog posts. Hope this helps, Scott
