What is Bingbot?
7 min read
Bingbot is the automated web crawler (bot) deployed by Microsoft to discover, fetch, render, and index webpages across the internet for Microsoft Bing, Microsoft Copilot, and syndicated partners including OpenAI’s ChatGPT Search, Yahoo, and DuckDuckGo. Launched in October 2010 to replace msnbot, modern Bingbot operates using an evergreen Microsoft Edge (Chromium) rendering engine and prioritizes crawl efficiency via IndexNow and HTTP 304 Not Modified conditional requests.
What is Bingbot & What Does It Crawl?
Every time a user searches on Bing, DuckDuckGo, Yahoo, or asks Microsoft Copilot / ChatGPT a question grounded in Bing’s web index, the results come from the searchable database maintained by Bingbot.
In 2019-2020, Microsoft upgraded Bingbot from an older Trident/EdgeHTML engine to a modern, evergreen Chromium-based Microsoft Edge rendering engine, aligning its HTML5, CSS, and ES6 JavaScript capabilities with Googlebot. Microsoft operates several specialized user-agent strings under the Bingbot family:
bingbot(Desktop & Mobile Web Crawler): The primary spider that fetches HTML documents for Bing’s core search index and AI grounding.AdIdxBot(Bing Ads Crawler): Inspects landing pages used in Microsoft Advertising PPC campaigns for quality and policy compliance.MicrosoftPreview: Generates live page previews and rich snapshots inside Microsoft products.
The #1 Technical SEO Bug: Why Firewalls Accidentally Block Bingbot (And Kill ChatGPT Citations)
One of the most damaging technical SEO bugs we uncover during B2B SaaS audits is when a website ranks well on Google, but has zero pages indexed in Bing and zero citations in ChatGPT because its CDN or Web Application Firewall (WAF) is blocking Bingbot!
How to Diagnose & Fix a WAF Block on Bingbot & AI Crawlers
Why It Happens: Security teams often enable aggressive "Bot Fight Mode" or ASN rate-limiting rules in Cloudflare, AWS WAF, or Akamai. If a firewall rule blocks Microsoft Azure data center IP ranges or challenges non-Google bots with a JavaScript CAPTCHA, Bingbot (along with OAI-SearchBot and ChatGPT-User) receives an HTTP 403 Forbidden status code and drops your site from the index.
How to Verify & Fix It:
- Open Bing Webmaster Tools → URL Inspection → click Live Test on your homepage and top BOFU pages. Confirm it returns
HTTP 200 OK. - In Cloudflare WAF, create an explicit Skip / Allow rule for Known Bots (
cf.client.bot eq true) and verified AI search crawlers (bingbot,OAI-SearchBot,ChatGPT-User,PerplexityBot). - Inspect your server access logs for
"bingbot/2.0"to verify clean200and304responses.
Bingbot vs. Googlebot vs. OAI-SearchBot: Crawler Comparison
| Crawler Attribute | Googlebot |
Bingbot |
OAI-SearchBot (OpenAI) |
|---|---|---|---|
| Owner & Ecosystem | Google (Google Search & Gemini AI Overviews). | Microsoft (Bing, Copilot, DuckDuckGo, Yahoo & ChatGPT Index). | OpenAI (Dedicated live search crawler for ChatGPT Search). |
| JavaScript Execution | Full two-wave headless Chromium rendering. | Supports Chromium JS, but strongly prefers SSR HTML to conserve crawl budget. | Raw HTML only (Does NOT execute client-side JavaScript/AJAX). |
| Push Indexing Support | XML Sitemaps (No IndexNow support). | Native IndexNow + XML Sitemaps. | Discovers URLs via Bing Index + direct web links. |
| Reverse DNS Hostname | *.googlebot.com or *.google.com |
*.search.msn.com |
Published IP ranges at openai.com/searchbot.json |
How to Optimize Your Website for Maximum Bingbot Crawl Efficiency
Microsoft’s engineering documentation emphasizes that Bingbot allocates crawl budget based on page freshness, authority, and server response efficiency. Follow these three rules to ensure Bingbot indexes 100% of your site:
- 1. Support HTTP
304 Not ModifiedCaching Headers: Configure your web server to return properETagandLast-Modifiedheaders so Bingbot receives an instant 304 Not Modified response on unchanged pages, freeing crawl budget for new articles. - 2. Keep Clean, Segmented XML Sitemaps: Separate your sitemaps into logical sub-sitemaps (pages, posts, glossary, images) with accurate
<lastmod>dates and zero 301 or 404 URLs. - 3. Never Spoof or Cloaking-Redirect Bingbot: Serving different HTML content to
bingbotthan to human visitors (cloaking) is a severe violation of Bing Webmaster Guidelines that results in full domain de-indexing.
Frequently Asked Questions
How can I verify if a request claiming to be Bingbot is real or a fake scraper?
Because malicious scrapers frequently fake the "bingbot/2.0" user-agent string, Microsoft recommends verifying Bingbot in two ways: (1) run a reverse DNS lookup (host IP) to confirm the hostname ends in .search.msn.com, followed by a forward DNS lookup to verify the IP matches, or (2) check the IP against Microsoft’s official daily JSON list at bing.com/toolbox/bingbot.json.
Should I use Crawl-delay in robots.txt for Bingbot?
Only if your origin server is severely underpowered and experiencing CPU spikes. Unlike Googlebot (which ignores Crawl-delay), Bingbot honors the Crawl-delay directive in robots.txt: meaning setting "Crawl-delay: 10" forces Bingbot to wait 10 seconds between every single request, which can drastically slow down how fast your pages get indexed in Bing and ChatGPT.
Does Bingbot execute client-side React or Vue JavaScript?
While evergreen Bingbot uses a headless Microsoft Edge Chromium browser capable of executing JavaScript, Microsoft’s official documentation states that rendering JavaScript at scale is resource-intensive and recommends Server-Side Rendering (SSR) or static HTML to guarantee complete indexing.
How do I request Bingbot to recrawl a page immediately after I update it?
You can either trigger an automated IndexNow API ping (which alerts Bingbot in real time) or paste the URL into the URL Inspection tool inside Bing Webmaster Tools and click "Request Indexing."