Search Results

Search found 14 results on 1 pages for 'bingbot'.

Page 1/1 | 1 

• Bingbot requests from Google IP address

  - by JITHIN JOSE

  We have some suspicious requests to our server, 74.125.186.46 - - [24/Aug/2014:23:24:11 -0500] "GET <url> HTTP/1.1" 200 16912 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 74.125.187.193 - - [24/Aug/2014:23:24:12 -0500] "GET <url> HTTP/1.1" 200 20119 "-" "Mozilla/5.0 (compatible; bingbot/2.0;…

  Read the article

• Bingbot seems to be adding "ForceRecrawl: 0" to URLs when crawling my sites

  - by Louis Somers

  I'm seeing this in the iis-logs of two websites that I maintain: GET /an/existing/page/on/my/site+ForceRecrawl:+0 - 80 - 207.46.195.105 HTTP/1.1 Mozilla/5.0+(compatible;+bingbot/2.0;++http://www.bing.com/bingbot.htm) I get about one or two of these per day from these IP addresses: 207.46.195.105, 65.52.110.190.. an more, all…

  Read the article

• Why deny access to website for msnbot/bingbot?

  - by Quandary

  I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for hacking and spamming. But why are there so many sites recommending to ban msnbot/bingbot? Since it's a search engine, even if only…

  Read the article

• Will bing bot index pages with invalid SSL certificates?

  - by Martin

  Bingbot and Yahoo slurp do not support SNI(Server Name Indication when using SSL). Ignoring other workarounds (multi domain certificates, non-SSL content etc.), will Bingbot index pages that have an invalid SSL certificate, eg. issued for example.net, but used on example.com? If possible please provide an example from Yahoo…

  Read the article

• What bots are really worth letting onto a site?

  - by blunders

  Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering if the goal of the site allowing bots is for the potential for the bot to send real traffic back to the site if there is any reason to allow bots that are not known to be sending real traffic back, and how…

  Read the article

• Server overhead caused by bots?

  - by giuseppe

  I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by the ajax link being placed on the left side of the website. The website is hosted by a VPS with 3 proc and 2GB RAM, with…

  Read the article

• Do web crawlers/spiders index azure web sites?

  - by Clay Shannon

  For somebody who wants their web site to be as discoverable as possible (and who doesn't?), are Microsoft's Azure web sites (azurewebsites.net) a feasible domain to host sites? I have a site that is both on an azurewebsites.net and hosted under a completely different name by discountasp.net Both of these sites are…

  Read the article

• Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go

  - by Glenn Plas

  After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots not respecting the robots.txt file (even after downloading it several times). Specifically Googlebot. It will support…

  Read the article

• Is there a way to disallow only crawling in https in robots.txt?

  - by David Wilkins

  I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only? According to Wikipedia, each protocol has its own robots.txt And according to Google's Robots.txt…

  Read the article

• Is there a way to disallow crawling of only HTTPS in robots.txt?

  - by David Wilkins

  I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only? According to Wikipedia, each protocol has its own robots.txt And according to Google's Robots.txt…

  Read the article

• Can a client dictate whether or not HttpContext is created?

  - by Keivan

  We are getting a lot of hits from Googlebot and BingBot and it appears that none of these requests have an HttpContext. I originally thought that every http request will get a context which obviously is not the case so I'm trying to understand how does an HttpContext gets constructed, is it part of…

  Read the article

• How can I avoid a 302 for Fetch as Bot?

  - by CookieMonster

  I originally posted this on Stackoverflow, but I believe here is a better place to ask. My web application is very similar to notepad.cc which redirects to a randomly generated URL upon access, e.g. http://myapp.com/roTr94h4Gd. (Please note that notepad.cc is not my site.) Probably because of this…

  Read the article

• Is a 302 redirect to a random URL from the homepage an SEO problem?

  - by CookieMonster

  I originally posted this on Stackoverflow, but I believe here is a better place to ask. My web application is very similar to notepad.cc which redirects to a randomly generated URL upon access, e.g. http://myapp.com/roTr94h4Gd. (Please note that notepad.cc is not my site.) Probably because of this…

  Read the article

• Why am I experiencing random connection timeouts? (CentOS)

  - by Ryan

  I have a CentOS server setup that currently hosts several websites (all relative of each other in some form or another). As of recently throughout the day at the most random times the website speed will lag to a crawl and eventually hit a connection timeout. When I say random times this typically…

  Read the article

1