Which token from a long User-Agent should I use in robots.txt?
- by Gaia
The definition of User-Agent states that several tokens can be included, as deemed necessary by the client.
I want to block certain bots via robots.txt and I am confused as to which part of the User-Agent string to use, especially for more obscure bots. For example:
Mozilla/5.0 (compatible; uMBot-LN/1.0; mailto:…