Does the user agent in any regular browser contain 'bot' or 'crawl'?

Posted by Echo on Pro Webmasters See other posts from Pro Webmasters or by Echo
Published on 2010-10-11T19:07:35Z Indexed on 2014/06/12 15:42 UTC
Read the original article Hit count: 189

Filed under:

user-agent

I check the user agent on my site to see if it is coming from a bot or not. If it is, I can do some little optimizations since they don't login. (I don't change the content at all)

After adding checks for 30-40+ bots, I'm getting tired of added them. So I was wondering if checking if it just contains 'bot' or 'crawl'. I know that wont get all bots, but it would get a lot of them. But if that could cause any false positives, then it would totally mess up the ability to add to cart, place an order, and login in.

Related posts about user-agent

What does it mean when a User-Agent has another User-Agent inside it?

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Basically, sometimes the user-agent will have its normal user-agent displayed, then at the end it will have teh "User-Agent: " tag displayed, and right after it another user-agent is shown. Sometimes, the second user-agent is just appended to the first one without the "User-Agent: " tag. Here are… >>> More
blocking bad bots with robots.txt in 2012 [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
does it still work good? I have this: # Generated using http://solidshellsecurity.com services # Begin block Bad-Robots from robots.txt User-agent: asterias Disallow:/ User-agent: BackDoorBot/1.0 Disallow:/ User-agent: Black Hole Disallow:/ User-agent: BlowFish/1.0 Disallow:/ User-agent: BotALot Disallow:/ User-agent:… >>> More
JavaScript - get detailed information about the browser

as seen on Stack Overflow - Search for 'Stack Overflow'
Basically I'm looking for something to give me easy access to information like useragentstring.com, but in JS, without me parsing the user agent and looking for each possible bit of text. The object could be something like this: browser = UserAgent.Browser; // Chrome browserVer… >>> More
User Agent in http client Android

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi everybody, I building an Https Client to send some data to a server, but I don't know what to pass in User-Agent. Should I use the webkit one's or do I have to build one explicitely for my App? I'm using this handy post Thanks for any help >>> More
Urlscan 3.1 block User Agent

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to block requests from certain User Agents to our Sharepoint Environment that have been identified after going through the IIS logs. I have tried the below by amending the urlscan.ini config file and doing and iisreset, but it doesn't block anything. Am I entering the correct strings? I'm… >>> More

Developer IT