How to disallow indexing but allow crawling?
- by John Doe
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. But if I do this (in robots.txt), would it still be crawled?
I mean, the full articles would be still reached by the crawler even though I disallowed the only page that links to them?
I don't want the webcrawler not to access the page and enter the links in them, but I just don't want it to save the information (that will be repeated in the full articles).