How to test robots.txt in googlebot to find out what is being indexed
Posted
by
Amar Jarubula
on Pro Webmasters
See other posts from Pro Webmasters
or by Amar Jarubula
Published on 2011-11-18T18:13:41Z
Indexed on
2011/11/20
10:25 UTC
Read the original article
Hit count: 291
This question is a continuation for this answer
How to check if googlebot will index a given url?
As was told I did go to the Webmaster Tools and tested contents of my robots.txt file. However this is just giving me the info if that content is good enough or not. However for my scenario I need to test whether disallowing some patterns is being indexed or not. For example I have something like this below in my robots.txt
disallow:/pattern*
My understanding is the URLs with word pattern should not crawled, but how do I test this pattern is enforced while indexing the website?
© Pro Webmasters or respective owner