Cross-submission robots.txt for multiple domains on single host

Posted by sidd.darko on Server Fault See other posts from Server Fault or by sidd.darko
Published on 2011-01-20T21:46:28Z Indexed on 2011/02/26 7:26 UTC
Read the original article Hit count: 557

Filed under:

iis7

|

robots.txt

|

sitemap

We are running a site with multiple languages hosted in a single environment on IIS7. For example,

oursite.com - english
oursite.de - german
oursite.es - spanish

This is a single-host environment. All of these sites are in the same application space on the same physical machine.

I need to do cross-submission of sitemaps via robots.txt. Looking at the sitemap.org guidelines for this suggest this is possible, but the example indicates different physical machines.

Will the following entries in oursite.com/robots.txt work?

http://www.oursite.com/sitemap-oursite-de.xml
http://www.oursite.com/sitemap-oursite-es.xml

© Server Fault or respective owner

Related posts about iis7

Configuring IIS7 404 page when using IIS7 urlrewrite module

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I've got custom errors to work for .aspx page like: www.domain.com/whateverdfdgdfg.aspx But, when no .aspx url is requested, (like http://www.domain.com/hfdkfdh4545) it results in an error: HTTP Error 404.0 - Not Found The resource you are looking for has been removed, had its name changed… >>> More
How to create a Subdomain in IIS7?

as seen on Server Fault - Search for 'Server Fault'
In IIS7 I have a site: http://www.mydomain.com/mysite How can I get the same site to appear as: http://mysite.mydomain.com ? I already have the DNS set up and can ping it, I just do not know how to configure it in IIS7. >>> More
How can I control which IP address IIS7 uses?

as seen on Server Fault - Search for 'Server Fault'
In Win2k3 I used httpcfg to tell IIS to listen to specific IP addresses on the server. I want to run Apache with VisualSVN Server on port 80 on another IP address but IIS7 binds to all ports by default. What utility for IIS7 controls the IIS7 bindings? Update: I found the answer. There is a utility… >>> More
WCF service hosted in IIS7 with administrator rights?

as seen on Server Fault - Search for 'Server Fault'
Hello, How do I grant administrator rights to a running WCF service hosted in IIS7? The problem is, my code works fine in a test console application runned as an administrator, but the same code used from WCF service in IIS7 fails. When I run the same console test application without admin rights… >>> More
authentication problem on windows vista (cookie is getting written but httpcontext.current.user is n

as seen on Stack Overflow - Search for 'Stack Overflow'
authentication problem on windows vista (iis7) (cookie is getting written but httpcontext.current.user is null) >>> More

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More