How to disallow indexing but allow crawling?
- by John Doe
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. But if I do this (in robots.txt), would it still be crawled?
I mean, the full articles would be still reached by the crawler even though I disallowed…