What is a good robots.txt for WP ?
- by Steven
What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress: /%category%/%postname%/.
My robots.txt currently looks like this (copied from somewhere a long time ago):
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow:…