How to test robots.txt in googlebot to find out what is being indexed
- by Amar Jarubula
This question is a continuation for this answer
How to check if googlebot will index a given url?
As was told I did go to the Webmaster Tools and tested contents of my robots.txt file. However this is just giving me the info if that content is good enough or not. However for my scenario I need to test whether disallowing some patterns is being indexed or not. For example I have something like this below in my robots.txt
disallow:/pattern*
My understanding is the URLs with word pattern should not crawled, but how do I test this pattern is enforced while indexing the website?