Short answer: A request that says “Googlebot” in its user-agent is not necessarily from Google. Verify the connecting IP with a reverse DNS lookup and a forward lookup, or match it against the correct Google-published IP range list. Never make a firewall decision from the user-agent string alone.
This guide is for site owners and SEOs investigating bot traffic in server or CDN logs. If you simply need examples of crawler names, see our Google user-agent reference. If a page is not appearing in search, use the noindex guide to check page directives separately; a verified crawl does not prove indexing.
What to collect before checking a request
Record the actual connecting client IP, timestamp, request path, HTTP status, and user-agent from the server or trusted CDN log. Behind a reverse proxy, the origin may see the proxy IP instead of the visitor IP: use the proxy’s authenticated client-IP field, not an arbitrary untrusted X-Forwarded-For header. Keep personal data handling and log retention appropriate to your site.
Method 1: Reverse DNS, then forward DNS
- Take the client IP from the log and perform a reverse DNS lookup:
host "$IP"(ordig -x "$IP" +short). - Check the hostname against the specific crawler category in Google’s verification documentation. Common crawlers use a
googlebot.comhostname pattern; other Google fetchers may have different patterns. Do not accept a lookalike suffix such asgooglebot.com.example.net. - Resolve the returned hostname forward:
host "$HOSTNAME". Confirm that its A or AAAA records include the original client IP. A reverse lookup alone is not sufficient.
For example, set IP to the logged address, run host "$IP", set HOSTNAME to the returned name and run host "$HOSTNAME". Compare the result to the same IP. If either lookup fails or the records disagree, treat the identity as unverified rather than assuming maliciousness.
Method 2: Match Google’s published IP ranges
Google also publishes JSON lists of IP prefixes for common crawlers, special-case crawlers and user-triggered fetchers. Use a maintained parser with IPv4/IPv6 CIDR matching, select the list appropriate to the claimed fetcher, and refresh the data periodically. An exact IP string search is not CIDR matching. Google’s verification guide is the source of truth for categories and current list URLs.
| Signal | What it proves | What it does not prove |
|---|---|---|
| Googlebot user-agent | The client claimed a name | The client is Google |
| Reverse + forward DNS | The IP resolves to an appropriate Google hostname and back | That the URL is indexed |
| CIDR match | The IP is inside a published range for that class | That the HTTP response was indexable |
Common mistakes in bot verification
- Trusting a user-agent: it is client-supplied text and can be copied.
- Checking only PTR: always forward-resolve the returned hostname too.
- Using the wrong Google list: common crawlers, special-case crawlers and user-triggered fetchers have different purposes and ranges.
- Reading the wrong IP behind a CDN: validate the client-IP source before applying DNS or range checks.
- Equating crawling with indexing: Search Console’s URL Inspection shows Google’s known index state, whereas a log entry only shows a request.
FAQ
Can a bot spoof Googlebot?
Yes. A user-agent is only a declared string. Verify the IP against Google’s documented DNS pattern or published ranges before granting special access.
Does a verified Googlebot request mean my page is indexed?
No. A crawl can still encounter a noindex directive, an inaccessible page, a redirect or a different canonical. Inspect the specific URL in Search Console and compare its live response.
Should I block all unknown bots?
That depends on your security policy and evidence. Do not automatically label failed DNS verification as proof of abuse; investigate the client, rate and behavior before changing firewall rules.
Sources and editorial note (checked September 2026): Google: Verify requests from crawlers and fetchers; Google: URL Inspection tool. Commands are illustrative; run them on an IP from your own logs. Google’s published ranges can change.