What bots are really worth letting onto a site?

Posted by blunders on Pro Webmasters See other posts from Pro Webmasters or by blunders
Published on 2014-06-13T04:42:43Z Indexed on 2014/06/13 15:43 UTC
Read the original article Hit count: 440

Filed under:

seo

|

web-crawlers

|

robots.txt

|

googlebot

|

bingbot

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering if the goal of the site allowing bots is for the potential for the bot to send real traffic back to the site if there is any reason to allow bots that are not known to be sending real traffic back, and how to spot these "good" bots; based on how they ID themselves, IPs they come from, behaviors, etc.

© Pro Webmasters or respective owner

Related posts about seo

Do i need a full time seo employee? [closed]

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I need seo done for 1 site that's still new. It's a niche listings/ecommerce site. And i need to make a hiring decision for this - whether there's a need for a full time employee dedicated to seo or is part time or a contract sufficient? Here's some info and assumptions : 1)The site's dynamic -… >>> More
1 SEO Article Vs 5 SEO Articles?

as seen on Ezine Articles - Search for 'Ezine Articles'
You've probably clicked on this article because you think it's a interesting article but the answer seems very easy or straight forward. If you think 5 SEO articles wins or is more beneficial to your business, then your like me. >>> More
SEO Training OR SEO Outsourcing

as seen on Ezine Articles - Search for 'Ezine Articles'
There is a lot of focus on outsourcing search optimization work, but this may not always be the best option. This article looks at why SEO Training can often be a better option because it results in more unique content which is better for search engines. >>> More
Get Professional SEO Service From SEO Experts

as seen on Ezine Articles - Search for 'Ezine Articles'
SEO service is imperative to online business. Business today has gone digital. With its power to perform international promotion, almost every brand is on a bid to establish an online presence. However, it's not easy to gain online presence without proper SEO service. >>> More
The Chinese SEO in the Internet SEO Formula

as seen on Ezine Articles - Search for 'Ezine Articles'
Most ecommerce websites are designed in order to drive the Chinese internet users to the website thus delivering a good user experience that converts the users into customers. According to a statistical report on the internet development in China, conducted during the year 2009, the Chinese SEO of… >>> More

Related posts about web-crawlers

Discover How Affiliates Marketers Can Optimize Their Blog For Search Engine Web Crawlers - #1

as seen on Ezine Articles - Search for 'Ezine Articles'
Search Engine Optimization is a FREE way of internet marketing that all internet marketers should take full advantage of it. It is all about building links to your websites or blogs. There are 2 ways of doing it. >>> More
How do web crawlers affect site statistics?

as seen on Stack Overflow - Search for 'Stack Overflow'
What are ways in which web crawlers (both from search engines and non-search engines) could affect site statistics (e.g., when doing AB-testing different page variations)? And what are ways to take care of these problems? For example: Do a lot of people writing web crawlers often delete their cookies… >>> More
Is there a list of known web crawlers?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know. Is there somewhere… >>> More
Detecting 'stealth' web-crawlers

as seen on Stack Overflow - Search for 'Stack Overflow'
What options are there to detect web-crawlers that do not want to be detected? (I know that listing detection techniques will allow the smart stealth-crawler programmer to make a better spider, but I do not think that we will ever be able to block smart stealth-crawlers anyway, only the ones that… >>> More
How to best develop web crawlers

as seen on Stack Overflow - Search for 'Stack Overflow'
Heyall, I am used to create some crawlers to compile information and as I come to a website I need the info I start a new crawler specific for that site, using shell scripts most of the time and sometime PHP. The way I do is with a simple for to iterate for the page list, a wget do download it and… >>> More