Is there a way to disallow crawling of only HTTPS in robots.txt?
- by David Wilkins
I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https only?
According to Wikipedia, each protocol has its own robots.txt
And according to Google's Robots.txt Specification, the robots.txt applies to http AND https
I don't want to Disallow: / for Bing totally, just over https.