robots.txt, how effective is it and how long does it take?
Posted
by
Stefan
on Pro Webmasters
See other posts from Pro Webmasters
or by Stefan
Published on 2012-07-09T22:18:15Z
Indexed on
2012/07/10
3:23 UTC
Read the original article
Hit count: 252
robots.txt
We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php.
When you search the company on engines such as Google, you get the site and a listing of its sub pages which now lead to outdated pages.
Our plan doesn't allow us to edit the .htaccess and the old pages are .html docs so I cannot use PHP redirects either.
So if I put in place a robots.txt telling the engines to not crawl beyond index.php, how effective will this be in preventing/removing crawled sub pages.
And rough guess, how long before the search engines would update?
© Pro Webmasters or respective owner