What is AhrefsBot?
6 min read
AhrefsBot is the automated commercial web crawler operated by the Ahrefs SEO intelligence suite (and its web search engine, Yep.com). Ranked by Cloudflare Radar among the top three most active bots on the internet, AhrefsBot crawls over 8 billion webpages every 24 hours and updates its live backlink index every 15 to 30 minutes. While AhrefsBot strictly obeys robots.txt directives (including Crawl-delay), blocking it on a B2B SaaS domain erases your public Domain Rating (DR) and organic visibility metrics in third-party SEO tools.
What is AhrefsBot & How Does It Work?
According to official Ahrefs technical documentation (ahrefs.com/robot), AhrefsBot continuously traverses the global web graph to discover new URLs, verify live and broken backlinks, calculate Domain Rating (DR) and URL Rating (UR), and power the index of Yep.com (Ahrefs’ revenue-sharing general search engine).
Ahrefs actually operates two distinct user-agents that appear in your server access logs:
AhrefsBot/7.0: The global index crawler that discovers external links and updates Ahrefs Site Explorer.AhrefsSiteAudit/6.1: The on-demand technical audit crawler triggered when a verified site owner or SEO agency runs an Ahrefs Site Audit project on a domain.
Should You Block or Allow AhrefsBot on a B2B SaaS Website?
System administrators sometimes block commercial SEO crawlers in Cloudflare or robots.txt after noticing thousands of requests in server logs. Before blocking AhrefsBot, weigh the strategic trade-offs for a B2B SaaS company:
| Impact Area | If You ALLOW AhrefsBot (Recommended for SaaS) | If You BLOCK AhrefsBot in Robots.txt / WAF |
|---|---|---|
| Google Search Rankings | Neutral (Google uses Googlebot, not AhrefsBot). | Neutral (Does not directly impact Google rankings). |
| Public Domain Rating (DR) & Link Data | Your full backlink graph, internal links, and DR remain accurate and up to date. | Ahrefs stops crawling your pages; outbound links disappear and Site Audit breaks. |
| Link Building & Digital PR Partnerships | Partners, publishers, and SaaS link builders can verify your site’s authority and live links. | Partners refuse link collaborations because Ahrefs flags your domain as uncrawlable. |
| Server Bandwidth & CPU Load | Light to Moderate (Easily throttled via Crawl-delay). |
Zero requests from Ahrefs IP ranges. |
How to Throttle AhrefsBot Using Crawl-Delay (Without Blocking It)
Unlike Googlebot (which completely ignores the non-standard Crawl-delay directive in robots.txt), AhrefsBot officially honors Crawl-delay! If AhrefsBot is crawling your server too aggressively, you do not need to block it with Disallow: /. Instead, simply set a polite delay between requests:
Copy-Paste Robots.txt Directives for AhrefsBot
Option 1: Recommended for SaaS (Allow AhrefsBot, Throttle to 1 Request Every 5 Seconds):
User-agent: AhrefsBot
Crawl-delay: 5
Disallow: /app/
Disallow: /api/
Option 2: Block Global AhrefsBot While Still Allowing Your Own Ahrefs Site Audits:
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Allow: /
How to Detect & Block Spoofed Fake AhrefsBot Scrapers
Malicious content scrapers and credential-stuffing bots frequently spoof the AhrefsBot/7.0 user-agent string to bypass basic security filters. Never block or whitelist AhrefsBot based solely on the user-agent text!
To verify whether a request comes from genuine Ahrefs infrastructure:
- Reverse DNS Lookup: Authentic AhrefsBot requests originate from hostnames ending in
*.ahrefs.comor*.ahrefs.net. - Official Published IP Ranges & Cloudflare Verified Bot Flag: Match the source IP against Ahrefs’ official JSON endpoint (
https://api.ahrefs.com/v3/public/crawler-ips) or use Cloudflare’s built-incf.client.botverified bot rule, which automatically validates AhrefsBot while blocking spoofed scrapers returning 403 Forbidden.
Frequently Asked Questions
Does blocking AhrefsBot in robots.txt hurt my Google rankings?
No. Google uses its own crawler (Googlebot) to index the web and calculate PageRank, so blocking AhrefsBot has zero direct impact on your Google Search rankings. However, it will break your data inside Ahrefs and prevent you from auditing your own site with Ahrefs tools.
If I block AhrefsBot on my website, will competitors still see my backlinks in Ahrefs?
Yes! Blocking AhrefsBot in your own robots.txt only stops AhrefsBot from crawling your website’s pages. Because your inbound backlinks live on external third-party websites (which allow AhrefsBot), Ahrefs will still discover and display every backlink pointing to your domain!
How can I slow down AhrefsBot if it is overloading my server?
Add a Crawl-delay directive specifically for AhrefsBot in your robots.txt file (for example, User-agent: AhrefsBot followed by Crawl-delay: 10). The value specifies the minimum number of seconds AhrefsBot must wait between consecutive HTTP requests to your server.
Why is Ahrefs Site Audit failing with a 403 Forbidden error on my site?
If Ahrefs Site Audit returns 403 Forbidden, your Web Application Firewall (such as Cloudflare Bot Fight Mode or Wordfence) is blocking Ahrefs IP ranges. Whitelist verified bots or allow the AhrefsSiteAudit user-agent and official IP list from api.ahrefs.com/v3/public/crawler-ips.