Google indexing page with parameters but page is Disallowed in robots.txt

Posted by Jakobud on Pro Webmasters See other posts from Pro Webmasters or by Jakobud
Published on 2012-09-14T15:57:15Z Indexed on 2012/09/14 21:48 UTC
Read the original article Hit count: 454

Filed under:

seo

|

google

|

robots.txt

I have the following in robots.txt:

User-agent: *
Disallow: /refer.php

User-agent: NinjaBot
Allow: /

Sitemap: http://www.mysite.com/sitemap.xml

The refer.php file does various things depending on what GET parameters are passed to it.

When I do a Google search, I see tons of results for pages like this:

http://www.mysite.com/refer.php?o=23945
http://www.mysite.com/refer.php?o=39858
http://www.mysite.com/refer.php?o=9683
http://www.mysite.com/refer.php?o=10569
http://www.mysite.com/refer.php?o=58304
http://www.mysite.com/refer.php?o=69604

Is the reason that Google is indexing these because I don't have an asterisk * after refer.php in the robots.txt ? Should changing it to Disallow: /refer.php* fix the problem?

© Pro Webmasters or respective owner

Related posts about seo

Do i need a full time seo employee? [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I need seo done for 1 site that's still new. It's a niche listings/ecommerce site. And i need to make a hiring decision for this - whether there's a need for a full time employee dedicated to seo or is part time or a contract sufficient? Here's some info and assumptions : 1)The site's dynamic -… >>> More
1 SEO Article Vs 5 SEO Articles?

as seen on Ezine Articles - Search for 'Ezine Articles'
You've probably clicked on this article because you think it's a interesting article but the answer seems very easy or straight forward. If you think 5 SEO articles wins or is more beneficial to your business, then your like me. >>> More
SEO Training OR SEO Outsourcing

as seen on Ezine Articles - Search for 'Ezine Articles'
There is a lot of focus on outsourcing search optimization work, but this may not always be the best option. This article looks at why SEO Training can often be a better option because it results in more unique content which is better for search engines. >>> More
Get Professional SEO Service From SEO Experts

as seen on Ezine Articles - Search for 'Ezine Articles'
SEO service is imperative to online business. Business today has gone digital. With its power to perform international promotion, almost every brand is on a bid to establish an online presence. However, it's not easy to gain online presence without proper SEO service. >>> More
The Chinese SEO in the Internet SEO Formula

as seen on Ezine Articles - Search for 'Ezine Articles'
Most ecommerce websites are designed in order to drive the Chinese internet users to the website thus delivering a good user experience that converts the users into customers. According to a statistical report on the internet development in China, conducted during the year 2009, the Chinese SEO of… >>> More

Related posts about google

Removing malware of a particular kind

as seen on Super User - Search for 'Super User'
I need to remove some malware from my computer. It is a trojan, and very annoying. It blocks access to Google and search sites. The trojan, with its name spelled out on each line cause it seems to block sites when i reference it in a url, is a r t (some text to mess it up) e m (more text i s First… >>> More
Trouble installing Matlab

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
So whenever I try to install Matlab, the first image appears, but then goes away and that's the end of it. Here's what I'm doing- user@host$> cd ~/mathworks_downloads user@host$> unzip matlab_R2012a_student_glnx86_installer.zip user@host$> ./install The Matlab logo and image pop up here… >>> More
Google chrome is always searching in local google domain instead of Google.com

as seen on Super User - Search for 'Super User'
I have changed in the searched preferences to google.com but still when I do search from the address bar (instant or non-instant) it will go to google.co.kr. Even though I change "Google.com in English", still same... The only way is to open google.com website first, then do search in it. So the… >>> More
Google I/O 2010: Google TV Keynote - Introducing Google TV

as seen on Google Code - Search for 'Google Code'
Google I/O 2010: Google TV Keynote - Introducing Google TV Due to licensing and permissions issues, we are unable to show the full Google TV demonstration from the Day 2 keynote at Google I/O. Until we are able to get these permissions, please check out these clips. For Google I/O session videos… >>> More
Google I/O 2010: Google TV Keynote - Android Apps On Google TV

as seen on Google Code - Search for 'Google Code'
Google I/O 2010: Google TV Keynote - Android Apps On Google TV Due to licensing and permissions issues, we are unable to show the full Google TV demonstration from the Day 2 keynote at Google I/O. Until we are able to get these permissions, please check out these clips. For Google I/O session videos… >>> More