Bingbot seems to be adding "ForceRecrawl: 0" to URLs when crawling my sites

Posted by Louis Somers on Pro Webmasters See other posts from Pro Webmasters or by Louis Somers
Published on 2011-10-26T22:32:11Z Indexed on 2011/11/27 2:02 UTC
Read the original article Hit count: 617

Filed under:

404

|

Bing

|

bing-webmaster-tools

I'm seeing this in the iis-logs of two websites that I maintain:

GET /an/existing/page/on/my/site+ForceRecrawl:+0 - 80 - 207.46.195.105 HTTP/1.1 Mozilla/5.0+(compatible;+bingbot/2.0;++http://www.bing.com/bingbot.htm)

I get about one or two of these per day from these IP addresses: 207.46.195.105, 65.52.110.190.. an more, all belonging to msnbot-ip.search.msn.com

Probably Microsoft has a bug in their crawler? Any way, doing a search on "ForceRecrawl: 0" in major search engines comes up with a bunch of random sites. Doing the search on StackOverflow or here gave no results (to my amazement). Am I the only one seeing this? I first noticed these on the 9th of this month, and I'm seeing them pass almost daily since...

Another thing that I think is crazy, is that the URL http://www.bing.com/bingbot.htm redirects to mail.live.com (hotmail).

Currently I'm returning 404's but I'm considering to catch these, strip the trailing " ForceRecrawl: 0" and process as if it were a legitimate url.

Could anyone shed some light on this? Could it have to do with some configuration or so in Bing's Webmaster Tools?

© Pro Webmasters or respective owner

Related posts about 404

ubuntu/apt-get update said "Failed to Fetch http:// .... 404 not found"

as seen on Server Fault - Search for 'Server Fault'
Hi all, I'm trying to run apt-get update on ubuntu 9.10 I've configured my proxy server and I can access the internet without any problem: /etc/apt# wget "http://www.google.com" Resolving (...) Proxy request sent, awaiting response... 200 OK Length: 292 [text/html] Saving to: `index.html' 100%[=================================================================================================================================>]… >>> More
Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
Protecting Apache with Fail2Ban

as seen on Pro Webmasters - Search for 'Pro Webmasters'
Having checked my Apache logs for the last two days I have noticed several attempts to access URLs such as /phpmyadmin, /phpldapadmin: 121.14.241.135 - - [09/Jun/2012:04:37:35 +0100] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 415 "-" "ZmEu" 121.14.241.135 - - [09/Jun/2012:04:37:35… >>> More
Django flatpages raising 404 when DEBUG is False (404 and 500 templates exist)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using Django 1.1.1 stable. When DEBUG is set to True Django flatpages works correctly; when DEBUG is False every flatpage I try to access raises a custom 404 error (my error template is obviously working correctly). Searching around on the internet suggests creating 404 and 500 templates which… >>> More
x.gif in Apache logs

as seen on Server Fault - Search for 'Server Fault'
I manage a Django site where we host the media on a subdomain. There shouldn't be any requests for media to the main domain. However I keep seeing these requests for "x.gif" showing up in the access logs on the domain that's handled by WSGI (not the media domain). Can anyone explain what this is… >>> More

Related posts about Bing

Integrate Bing Search API into ASP.Net application

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Couple of months back, I wrote an article about how to integrate Bing Search engine (API 2.0) with ASP.Net website. You can refer the article here http://weblogs.asp.net/sreejukg/archive/2012/04/07/integrate-bing-api-for-search-inside-asp-net-web-application.aspx Things are changing rapidly… >>> More
Bing Desktop Automatically Downloads Bing Wallpapers to Your Computer

as seen on How to geek - Search for 'How to geek'
Windows 7: Bing Desktop is a new and lightweight offering from Microsoft that automatically swaps your desktop background every day and offers quick access to the Bing search engine. In addition to downloading the Bing wallpaper, Bing Desktop also includes a small search box that allows you to search… >>> More
add shapes to bing maps from locations stored in a database (bing maps ajax control)

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to use the Bing Maps Ajax Control to plot pins of locations stored in a database to the bing map on a web page. All the locations are geocoded and the lat longs stored in the database. I am using ASP.NET (C#), but can't figure out or find any tutorials on how to go about doing this. All… >>> More
Introducing the Earthquake Locator – A Bing Maps Silverlight Application, part 1

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Update: Live demo and source code now available! The recent wave of earthquakes (no pun intended) being reported in the news got me wondering about the frequency and severity of earthquakes around the world. Since I’ve been doing a lot of Silverlight development lately, I decided to scratch my curiosity… >>> More
Bing Makes Its iPhone Debut

as seen on Internet.com - Search for 'Internet.com'
Microsoft reaches out to Apple's iPhone to extend Bing's reach. >>> More