Skip to answers
Knowledge BaseTraffic quality & conversion answersTechnical documentation ↗
Browse answers

Traffic Detective

Is this Googlebot, an AI crawler or someone pretending to be one?

In short

Inspect available crawler classification evidence and request patterns. A user-agent string alone cannot verify a crawler's identity or show whether a page was indexed.

Research context

Google documents forward-confirmed reverse DNS and published IP ranges for verifying its crawler requests. User-Agent text alone is not authentication; other crawlers need their own publisher-specific checks.

There are many different Googlebots. Some are simple, non-rendering request bots; others render pages in both desktop and mobile Chrome. So if you start blocking automation, they're among the first false positives you'll run into.

That's why you need at least IP verification, so that blocking datacenter traffic doesn't block Google's crawlers too.

AI crawlers are a different story. There are documented cases of custom stealth browsers being used to retrieve websites or API data.

So there isn't a clear-cut rule here either. We opted for a basic one:

If an AI company publishes its IP ranges, requests from those ranges are allowed. Otherwise, a request claiming to be that crawler is marked as inconsistent with its user agent.

Some AI crawlers, such as Cloudflare's crawler, we label as indecent AI crawlers. Cloudflare has positioned itself around guarding websites worldwide. We urge Cloudflare to reconsider its approach here.

Next step: Is someone scraping our products or prices? · Related guide: Protect the form and the site, not just the ad account

Sources and further reading

Button Solutions Knowledge Base Evidence before action.