In the past few weeks I've seen some reports that Microsoft Bing has indexing issues of its own. That said, Bing doesn't index pages for its search engine like it used to. In fact, I was told that Microsoft has confirmed the problem with the CloudFlare engineers and it needs to be fixed by Microsoft.
Some thought it had something to do with the fact that CloudFlare was blocked, but it could be more common than just that based on the reports I get both publicly and privately via email.
I asked Fabrice Canel at Microsoft about this and he responded on Twitter with a set of best practices that Bing wants to index. 🙂
Best practices for when a website isn't listed on Bing. First inspect it using the URL Inspection Tool https://t.co/InvjmzlbBB to identify potential crawling issues, then read the Bing Webmaster Guidelines https://t.co/Gbrm4B2XMX. Be sure to make sure your website complies with the rules
– Fabrice Canel (@facan) October 26, 2020
But let me walk you through what I've seen. A CloudFlare community thread has tons of complaints as of the last day. Here is the first complaint:
Hello, several people had the same problem as me, including that their Cloudflare hosted websites had blocked Microsoft IP addresses from which the WebCrawler was trying to access the websites.
I just checked my website email and received a notification from Microsoft Bing that the web crawler was unable to access my Cloudflare website.
The folks at WebmasterWorld noticed it too, here are some quotes from there:
And now for some really interesting information: I blocked this IP in the Ubuntu UFW firewall on one of my websites and after about a week it was de-indexed by BING. It seems that BING is a lot stricter and if they can't crawl your website they are taking drastic action! It could just be a coincidence. So it is really not that clear whether this is a "bad" IP, as the ratings of users of "Abusive IP" are not proof, other than guesswork on their part. There is a Bing deindexing update / bug. Many users reported this on other forums. Check with Bing that my website has been completely deindexed like a month ago. That's what I get because I never bothered checking out Bing. In September, one of my websites was banned from Bing's SERP. On October 23rd, I noticed my three other websites were blocked. Even if I type in "site: example.com", there are no results. Just the link "Some results have been removed"
CloudFlare confirmed the problem by saying, "Microsoft has acknowledged that this is a legitimate crawler and that this is a problem on their part. They are working on the solution (by setting it to match their published expected behavior)."
Microsoft Bing is apparently working on a fix.
I'm not sure if this is more than just a CloudFlare and Microsoft Bing issue or not, but there are many complaints about it.
The good news is that Microsoft Bing is apparently working on a fix.
Forum discussion in the CloudFlare Community.
Update: More from Fabrice Canel:
For Cloudflare, some websites reported that a small percentage of the Bing crawler had inadequate DNS resolution and we crawled outside of the usual IP addresses for a few days. We fixed that yesterday.
– Fabrice Canel (@facan) October 27, 2020