Search engine optimization (SEO) is important for any publically facing web-site. A large % of traffic
to sites now comes directly from search engines, and improving your site’s search relevancy will lead
to more users visiting your site from search engine queries. This can directly or indirectly increase
the money you make through your site. This blog post covers
how you can use
the free Microsoft URL Rewrite Extension
to fix a bunch of common SEO problems that your site might have. It takes less than 15 minutes (and no code changes)
to apply 4 simple URL Rewrite rules
to your site, and in doing so cause search engines
to drive more visitors and traffic
to your site.
The techniques below work equally well with both ASP.NET Web Forms and ASP.NET MVC based sites. They also works with all versions of ASP.NET (and even work with non-ASP.NET content). [In addition
to blogging, I am also now using Twitter for quick updates and
to share links. Follow me at: twitter.com/scottgu] Measuring
the SEO of your website with
the Microsoft SEO Toolkit A few months ago I blogged about
the free SEO Toolkit that we’ve shipped. This useful tool enables you
to automatically crawl/scan your site for SEO correctness, and it then flags any SEO issues it finds. I highly recommend downloading and using
the tool against any public site you work on. It makes it easy
to spot SEO issues you might have in your site, and pinpoint ways
to optimize it further. Below is a simple example of a report I ran against one of my sites (www.scottgu.com) prior
to applying
the URL Rewrite rules I’ll cover later in this blog post: Search Relevancy and URL Splitting Two of
the important things that search engines evaluate when assessing your site’s “search relevancy” are:
How many other sites link
to your content. Search engines assume that if a lot of people around
the web are linking
to your content, then it is likely useful and so weight it higher in relevancy.
The uniqueness of
the content it finds on your site. If search engines find that
the content is duplicated in multiple places around
the Internet (or on multiple URLs on your site) then it is likely
to drop
the relevancy of
the content. One of
the things you want
to be very careful
to avoid when building public facing sites is
to not allow different URLs
to retrieve
the same content within your site. Doing so will hurt with both of
the situations above. In particular, allowing external sites
to link
to the same content with multiple URLs will cause your link-count and page-ranking
to be split up across those different URLs (and so give you a smaller page rank than what it would otherwise be if it was just one URL). Not allowing external sites
to link
to you in different ways sounds easy in theory – but you might wonder what exactly this means in practice and
how you avoid it. 4 Really Common SEO Problems Your Sites Might Have Below are 4 really common scenarios that can cause your site
to inadvertently expose multiple URLs for
the same content. When this happens external sites linking
to yours will end up splitting their page links across multiple URLs - and as a result cause you
to have a lower page ranking with search engines than you deserve. SEO Problem #1: Default Document IIS (and other web servers) supports
the concept of a “default document”. This allows you
to avoid having
to explicitly specify
the page you want
to serve at either
the root of
the web-site/application, or within a sub-directory. This is convenient – but means that by default this content is available via two different publically exposed URLs (which is bad). For example: http://scottgu.com/ http://scottgu.com/default.aspx SEO Problem #2: Different URL Casings Web developers often don’t realize URLs are case sensitive
to search engines on
the web. This means that search engines will treat
the following links as two completely different URLs: http://scottgu.com/Albums.aspx http://scottgu.com/albums.aspx SEO Problem #3: Trailing Slashes Consider
the below two URLs – they might look
the same at first, but they are subtly different.
The trailing slash creates yet another situation that causes search engines
to treat
the URLs as different and so split search rankings: http://scottgu.com http://scottgu.com/ SEO Problem #4: Canonical Host Names Sometimes sites support scenarios where they support a web-site with both a leading “www” hostname prefix as well as just
the hostname itself. This causes search engines
to treat
the URLs as different and split search rankling: http://scottgu.com/albums.aspx/ http://www.scottgu.com/albums.aspx/
How to Easily Fix these SEO Problems in 10 minutes (or less) using IIS Rewrite If you haven’t been careful when coding your sites, chances are you are suffering from one (or more) of
the above SEO problems. Addressing these issues will improve your search engine relevancy ranking and drive more traffic
to your site.
The “good news” is that fixing
the above 4 issues is really easy using
the URL Rewrite Extension. This is a completely free Microsoft extension available for IIS 7.x (on Windows Server 2008, Windows Server 2008 R2, Windows 7 and Windows Vista).
The great thing about using
the IIS Rewrite extension is that it allows you
to fix
the above problems *without* having
to change any code within your applications. You can easily install
the URL Rewrite Extension in under 3 minutes using
the Microsoft Web Platform Installer (a free tool we ship that automates setting up web servers and development machines). Just click
the green “Install Now” button on
the URL Rewrite Spotlight page
to install it on your Windows Server 2008, Windows 7 or Windows Vista machine: Once installed you’ll find that a new “URL Rewrite” icon is available within
the IIS 7 Admin Tool: Double-clicking
the icon will open up
the URL Rewrite admin panel – which will display
the list of URL Rewrite rules configured for a particular application or site: Notice that our rewrite rule list above is currently empty (which is
the default when you first install
the extension). We can click
the “Add Rule…” link button in
the top-right of
the panel
to add and enable new URL Rewriting logic for our site. Scenario 1: Handling Default Document Scenarios One of
the SEO problems I discussed earlier in this post was
the scenario where
the “default document” feature of IIS causes you
to inadvertently expose two URLs for
the same content on your site. For example: http://scottgu.com/ http://scottgu.com/default.aspx We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates
to the second URL
to instead go
to the first one. We will setup
the HTTP redirect
to be a “permanent redirect” – which will indicate
to search engines that they should follow
the redirect and use
the new URL they are redirected
to as
the identifier of
the content they retrieve. Let’s look at
how we can create such a rule. We’ll begin by clicking
the “Add Rule” link in
the screenshot above. This will cause
the below dialog
to display: We’ll select
the “Blank Rule” template within
the “Inbound rules” section
to create a new custom URL Rewriting rule. This will display an empty pane like below: Don’t worry – setting up
the above rule is easy.
The following 4 steps explain
how to do so: Step 1: Name
the Rule Our first step will be
to name
the rule we are creating. Naming it with a descriptive name will make it easier
to find and understand later. Let’s name this rule our “Default Document URL Rewrite” rule: Step 2: Setup
the Regular Expression that Matches this Rule Our second step will be
to specify a regular expression filter that will cause this rule
to execute when an incoming URL matches
the regex pattern. Don’t worry if you aren’t good with regular expressions - I suck at them too.
The trick is
to know someone who is good at them or copy/paste them from a web-site. Below we are going
to specify
the following regular expression as our pattern rule: (.*?)/?Default\.aspx$ This pattern will match any URL string that ends with Default.aspx.
The "(.*?)" matches any preceding character zero or more times.
The "/?" part says
to match
the slash symbol zero or one times.
The "$" symbol at
the end will ensure that
the pattern will only match strings that end with Default.aspx. Combining all these regex elements allows this rule
to work not only for
the root of your web site (e.g. http://scottgu.com/default.aspx) but also for any application or subdirectory within
the site (e.g. http://scottgu.com/photos/default.aspx. Because
the “ignore case” checkbox is selected it will match both “Default.aspx” as well as “default.aspx” within
the URL. One nice feature built-into
the rule editor is a “Test pattern” button that you can click
to bring up a dialog that allows you
to test out a few URLs with
the rule you are configuring: Above I've added a “products/default.aspx” URL and clicked
the “Test” button. This will give me immediate feedback on whether
the rule will execute for it. Step 3: Setup a Permanent Redirect Action We’ll then setup an action
to occur when our regular expression pattern matches
the incoming URL: In
the dialog above I’ve changed
the “Action Type” drop down
to be a “Redirect” action.
The “Redirect Type” will be a HTTP 301 Permanent redirect – which means search engines will follow it. I’ve also set
the “Redirect URL” property
to be: {R:1}/ This indicates that we want
to redirect
the web client requesting
the original URL
to a new URL that has
the originally requested URL path - minus
the "Default.aspx" in it. For example, requests for http://scottgu.com/default.aspx will be redirected
to http://scottgu.com/, and requests for http://scottgu.com/photos/default.aspx will be redirected
to http://scottgu.com/photos/
The "{R:N}" regex construct, where N >= 0, is called a back-reference and N is
the back-reference index. In
the case of our pattern "(.*?)/?Default\.aspx$", if
the input URL is "products/Default.aspx" then {R:0} will contain "products/Default.aspx" and {R:1} will contain "products". We are going
to use this {R:1}/ value
to be
the URL we redirect users
to. Step 4: Apply and Save
the Rule Our final step is
to click
the “Apply” button in
the top right hand of
the IIS admin tool – which will cause
the tool
to persist
the URL Rewrite rule into our application’s root web.config file (under a <system.webServer/rewrite> configuration section): <configuration> <system.webServer> <rewrite> <rules> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Because IIS 7.x and ASP.NET share
the same web.config files, you can actually just copy/paste
the above code into your web.config files using Visual Studio and skip
the need
to run
the admin tool entirely. This also makes adding/deploying URL Rewrite rules with your ASP.NET applications really easy. Step 5: Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://scottgu.com/ http://scottgu.com/default.aspx Notice that
the second URL automatically redirects
to the first one. Because it is a permanent redirect, search engines will follow
the URL and should update
the page ranking of http://scottgu.com
to include links
to http://scottgu.com/default.aspx as well. Scenario 2: Different URL Casing Another common SEO problem I discussed earlier in this post is that URLs are case sensitive
to search engines on
the web. This means that search engines will treat
the following links as two completely different URLs: http://scottgu.com/Albums.aspx http://scottgu.com/albums.aspx We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates
to the first URL
to instead go
to the second (all lower-case) one. Like before, we will setup
the HTTP redirect
to be a “permanent redirect” – which will indicate
to search engines that they should follow
the redirect and use
the new URL they are redirected
to as
the identifier of
the content they retrieve.
To create such a rule we’ll click
the “Add Rule” link in
the URL Rewrite admin tool again. This will cause
the “Add Rule” dialog
to appear again: Unlike
the previous scenario (where we created a “Blank Rule”), with this scenario we can take advantage of a built-in “Enforce lowercase URLs” rule template. When we click
the “ok” button we’ll see
the following dialog which asks us if we want
to create a rule that enforces
the use of lowercase letters in URLs: When we click
the “Yes” button we’ll get a pre-written rule that automatically performs a permanent redirect if an incoming URL has upper-case characters in it – and automatically send users
to a lower-case version of
the URL: We can click
the “Apply” button
to use this rule “as-is” and have it apply
to all incoming URLs
to our site. Because my www.scottgu.com site uses ASP.NET Web Forms, I’m going
to make one small
change to the rule we generated above – which is
to add a condition that will ensure that URLs
to ASP.NET’s built-in “WebResource.axd” handler are excluded from our case-sensitivity URL Rewrite logic. URLs
to the WebResource.axd handler will only come from server-controls emitted from my pages – and will never be linked
to from external sites. While my site will continue
to function fine if we redirect these URLs
to automatically be lower-case – doing so isn’t necessary and will add an extra HTTP redirect
to many of my pages.
The good news is that adding a condition that prevents my URL Rewriting rule from happening with certain URLs is easy. We simply need
to expand
the “Conditions” section of
the form above We can then click
the “Add” button
to add a condition clause. This will bring up
the “Add Condition” dialog: Above I’ve entered {URL} as
the Condition input – and said that this rule should only execute if
the URL does not match a regex pattern which contains
the string “WebResource.axd”. This will ensure that WebResource.axd URLs
to my site will be allowed
to execute just fine without having
the URL be re-written
to be all lower-case. Note: If you have static resources (like references
to .jpg, .css, and .js files) within your site that currently use upper-case characters you’ll probably want
to add additional condition filter clauses so that URLs
to them also don’t get redirected
to be lower-case (just add rules for patterns like .jpg, .gif, .js, etc). Your site will continue
to work fine if these URLs get redirected
to be lower case (meaning
the site won’t break) – but it will cause an extra HTTP redirect
to happen on your site for URLs that don’t need
to be redirected for SEO reasons. So setting up a condition clause makes sense
to add. When I click
the “ok” button above and apply our lower-case rewriting rule
the admin tool will save
the following additional rule
to our web.config file: <configuration> <system.webServer> <rewrite> <rules> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> <rule name="Lower Case URLs" stopProcessing="true"> <match url="[A-Z]" ignoreCase="false" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{ToLower:{URL}}" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://scottgu.com/Albums.aspx http://scottgu.com/albums.aspx Notice that
the first URL (which has a capital “A”) automatically does a redirect
to a lower-case version of
the URL. Scenario 3: Trailing Slashes Another common SEO problem I discussed earlier in this post is
the scenario of trailing slashes within URLs.
The trailing slash creates yet another situation that causes search engines
to treat
the URLs as different and so split search rankings: http://scottgu.com http://scottgu.com/ We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates
to the first URL (that does not have a trailing slash)
to instead go
to the second one that does. Like before, we will setup
the HTTP redirect
to be a “permanent redirect” – which will indicate
to search engines that they should follow
the redirect and use
the new URL they are redirected
to as
the identifier of
the content they retrieve.
To create such a rule we’ll click
the “Add Rule” link in
the URL Rewrite admin tool again. This will cause
the “Add Rule” dialog
to appear again:
The URL Rewrite admin tool has a built-in “Append or remove
the trailing slash symbol” rule template. When we select it and click
the “ok” button we’ll see
the following dialog which asks us if we want
to create a rule that automatically redirects users
to a URL with a trailing slash if one isn’t present: Like within our previous lower-casing rewrite rule we’ll add one additional condition clause that will exclude WebResource.axd URLs from being processed by this rule. This will avoid an unnecessary redirect for happening for those URLs. When we click
the “OK” button we’ll get a pre-written rule that automatically performs a permanent redirect if
the URL doesn’t have a trailing slash – and if
the URL is not processed by either a directory or a file. This will save
the following additional rule
to our web.config file: <configuration> <system.webServer> <rewrite> <rules> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> <rule name="Lower Case URLs" stopProcessing="true"> <match url="[A-Z]" ignoreCase="false" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{ToLower:{URL}}" /> </rule> <rule name="Trailing Slash" stopProcessing="true"> <match url="(.*[^/])$" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" /> <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" /> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{R:1}/" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://scottgu.com http://scottgu.com/ Notice that
the first URL (which has no trailing slash) automatically does a redirect
to a URL with
the trailing slash. Because it is a permanent redirect, search engines will follow
the URL and update
the page ranking. Scenario 4: Canonical Host Names
The final SEO problem I discussed earlier are scenarios where a site works with both a leading “www” hostname prefix as well as just
the hostname itself. This causes search engines
to treat
the URLs as different and split search rankling: http://www.scottgu.com/albums.aspx http://scottgu.com/albums.aspx We can fix this by adding a new IIS Rewrite rule that automatically redirects anyone who navigates
to the first URL (that has a www prefix)
to instead go
to the second URL. Like before, we will setup
the HTTP redirect
to be a “permanent redirect” – which will indicate
to search engines that they should follow
the redirect and use
the new URL they are redirected
to as
the identifier of
the content they retrieve.
To create such a rule we’ll click
the “Add Rule” link in
the URL Rewrite admin tool again. This will cause
the “Add Rule” dialog
to appear again:
The URL Rewrite admin tool has a built-in “Canonical domain name” rule template. When we select it and click
the “ok” button we’ll see
the following dialog which asks us if we want
to create a redirect rule that automatically redirects users
to a primary host name URL: Above I’m entering
the primary URL address I want
to expose
to the web: scottgu.com. When we click
the “OK” button we’ll get a pre-written rule that automatically performs a permanent redirect if
the URL has another leading domain name prefix. This will save
the following additional rule
to our web.config file: <configuration> <system.webServer> <rewrite> <rules> <rule name="Cannonical Hostname"> <match url="(.*)" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{HTTP_HOST}" pattern="^scottgu\.com$" negate="true" /> </conditions> <action type="Redirect" url="http://scottgu.com/{R:1}" /> </rule> <rule name="Default Document" stopProcessing="true"> <match url="(.*?)/?Default\.aspx$" /> <action type="Redirect" url="{R:1}/" /> </rule> <rule name="Lower Case URLs" stopProcessing="true"> <match url="[A-Z]" ignoreCase="false" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{ToLower:{URL}}" /> </rule> <rule name="Trailing Slash" stopProcessing="true"> <match url="(.*[^/])$" /> <conditions logicalGrouping="MatchAll" trackAllCaptures="false"> <add input="{REQUEST_FILENAME}" matchType="IsDirectory" negate="true" /> <add input="{REQUEST_FILENAME}" matchType="IsFile" negate="true" /> <add input="{URL}" pattern="WebResource.axd" negate="true" /> </conditions> <action type="Redirect" url="{R:1}/" /> </rule> </rules> </rewrite> </system.webServer> </configuration> Try
the Rule Out Now that we’ve saved
the rule, let’s try it out on our site. Try
the following two URLs on my site: http://www.scottgu.com/albums.aspx http://scottgu.com/albums.aspx Notice that
the first URL (which has
the “www” prefix) now automatically does a redirect
to the second URL which does not have
the www prefix. Because it is a permanent redirect, search engines will follow
the URL and update
the page ranking. 4 Simple Rules for Improved SEO
The above 4 rules are pretty easy
to setup and should take less than 15 minutes
to configure on existing sites you already have.
The beauty of using a solution like
the URL Rewrite Extension is that you can take advantage of it without having
to change code within your web-site – and without having
to break any existing links already pointing at your site. Users who follow existing links will be automatically redirected
to the new URLs you wish
to publish. And search engines will start
to give your site a higher search relevancy ranking – which will list your site higher in search results and drive more traffic
to it. Customizing your URL Rewriting rules further is easy to-do either by editing
the web.config file directly, or alternatively, just double click
the URL Rewrite icon within
the IIS 7.x admin tool and it will list all
the active rules for your web-site or application: Clicking any of
the rules above will open
the rules editor back up and allow you
to tweak/customize/save them further. Summary Measuring and improving SEO is something every developer building a public-facing web-site needs
to think about and focus on. If you haven’t already, download and use
the SEO Toolkit
to analyze
the SEO of your sites today. New URL Routing features in ASP.NET MVC and ASP.NET Web Forms 4 make it much easier
to build applications that have more control over
the URLs that are published. Tools like
the URL Rewrite Extension that I’ve talked about in this blog post make it much easier
to improve
the URLs that are published from sites you already have built today – without requiring you
to change a lot of code.
The URL Rewrite Extension provides a bunch of additional great capabilities – far beyond just SEO - as well. I’ll be covering these additional capabilities more in future blog posts. Hope this helps, Scott
