Hi,

I'm a bit vague on the precise syntax of robots.txt, but what I'm trying to achieve is:

• Tell all user agents not to crawl certain pages
• Tell certain user agents not to crawl anything

(basically, some pages with enormous amounts of data should never be crawled; and some voracious but useless search engines, e.g. Cuil, should never crawl anything)

If I do something like this:

User-agent: *
Disallow: /path/page1.aspx
Disallow: /path/page2.aspx
Disallow: /path/page3.aspx

User-agent: twiceler
Disallow: /

..will it flow through as expected, with all user agents matching the first rule and skipping page1, page2 and page3; and twiceler matching the second rule and skipping everything?