Multiple SiteMap: entries in robots.txt?

Posted by user306942 on Stack Overflow See other posts from Stack Overflow or by user306942
Published on 2010-04-07T16:31:58Z Indexed on 2010/04/07 16:43 UTC
Read the original article Hit count: 270

Filed under:

sitemap

|

robots.txt

I have been searching around using Google but I can't find an answer to this question.

A robots.txt file can contain the following line:

Sitemap: http://www.mysite.com/sitemapindex.xml

but is it possible to specify MULTIPLE sitemap index files in the robots.txt and have the search engines recognize that and crawl ALL of the sitemaps referenced in each sitemap index file? For example, will this work:

Sitemap: http://www.mysite.com/sitemapindex1.xml

Sitemap: http://www.mysite.com/sitemapindex2.xml

Sitemap: http://www.mysite.com/sitemapindex3.xml

© Stack Overflow or respective owner

Related posts about sitemap

problem with Webmaster Google Sitemap

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a wp mu 3.6.1 with domain mapping (0.5.4.3) with w3tc (0.9.3) and Google XML Sitemaps (4.0 BETA). I have 4 different sitemaps. sub-1.com/sitemap.xml sub-2.com/sitemap.xml sub-3.com/sitemap.xml sub-4.com/sitemap.xml on google webmaster i got 59 errors & 14 warnings. Sitemap errorsErrors: We… >>> More
The Sitemap Paradox

as seen on Pro Webmasters - Search for 'Pro Webmasters'
We use a sitemap on Stack Overflow, but I have mixed feelings about it. Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using… >>> More
Sitemaps - do I need to submit each sitemap in sitemap_index.xml to Google Webmaster tools?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I am having a Wordpress blog on my CentOS server. There is no sitemap.xml in the root directory but there is sitemap_index.xml file in the root directory which contains the following code: <?xml-stylesheet type="text/xsl" href="http://mywebsite.com/wp-content/plugins/wordpress-seo/css/xml-sitemap-xsl… >>> More
Yahoo sitemap validation

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Hello, I am trying to submit sitemap.xml (index) to Yahoo Site Explorer but with no luck. I tried using website feed option in the site explorer to submit the sitemap, but I got validation errors. However, when submitting the same sitemap to google webmaster tools, the sitemap was validated successfully… >>> More
Modify code to change timestamp timezone in sitemap

as seen on Stack Overflow - Search for 'Stack Overflow'
Below is the code from a plugin I use for sitemaps. I would like to know if there's a way to "enforce" a different timezone on all date variable and functions in it. If not, how do I modify the code to change the timestamp to reflect a different timezone? Say for example, to America/New_York timezone… >>> More

Related posts about robots.txt

Robots.txt practices with .htaccess redirections (inherits)

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains and subdomains, including the primary domain, is redirected via htaccess 301s to their own subdirectories… >>> More
mod evasive not working properly on ubuntu 10.04

as seen on Server Fault - Search for 'Server Fault'
I have an ubuntu 10.04 server where I installed mod_evasive using apt-get install libapache2-mod-evasive I already tried several configurations, the result stays the same. The blocking does work, but randomly. I tried with low limis and long blocking periods as well as short limits. The behaviour… >>> More
Cross-domain jQuery using YQL gives robots.txt error

as seen on Stack Overflow - Search for 'Stack Overflow'
On the page http://qxlapps.dk/test.htm I am trying to perform an Ajax load from another domain, qxlapp.dk. I am using James Padolsey's xdomainajax.js plugin from: http://james.padolsey.com/javascript/cross-domain-requests-with-jquery/ When I open my test page, I get no output, but FireBug shows… >>> More
Asterisk in robots.txt

as seen on Stack Overflow - Search for 'Stack Overflow'
Wondering if following will work for google in robots.txt Disallow: /*.action I need to exclude all urls ending with .action. Is this correct? >>> More
SEO chaos from changing robots.txt file in Wordpress site

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I recently edited the robots.txt file in my site using a wordpress plugin. However, since i did this, google seems to have removed my site from their search page. I'd appreciate if I could get an expert opinion on why this is so, and a possible solution. I'd initially done it to increase… >>> More