Is my robots.txt working as it should?
Posted
by
TigerBlood
on Pro Webmasters
See other posts from Pro Webmasters
or by TigerBlood
Published on 2011-03-04T21:22:41Z
Indexed on
2011/03/04
23:33 UTC
Read the original article
Hit count: 309
robots.txt
I want crawlers to have access to http://www.example.com but not http://www.example.com/
My robots.txt is as follows:
User-agent: *
Allow: /$
Disallow: /
My site is in google search results, but I am not coming up in Bing, Yahoo, etc. I have had the same robots.txt since last year, and I initially requested inclusion ~1 year ago, having also resubmitted the URL to those latter search engines several times since as well.
Is my robots.txt blocking those other crawlers? And if so, why not google as well?
Thanks in advance!
© Pro Webmasters or respective owner