Google indexing page with parameters but page is Disallowed in robots.txt
- by Jakobud
I have the following in robots.txt:
User-agent: *
Disallow: /refer.php
User-agent: NinjaBot
Allow: /
Sitemap: http://www.mysite.com/sitemap.xml
The refer.php file does various things depending on what GET parameters are passed to it.
When I do a Google search, I see tons of results for pages like this:
http://www.mysite.com/refer.php?o=23945
http://www.mysite.com/refer.php?o=39858
http://www.mysite.com/refer.php?o=9683
http://www.mysite.com/refer.php?o=10569
http://www.mysite.com/refer.php?o=58304
http://www.mysite.com/refer.php?o=69604
Is the reason that Google is indexing these because I don't have an asterisk * after refer.php in the robots.txt ? Should changing it to Disallow: /refer.php* fix the problem?