Google User Agents & Googlebot Verification Guide

A Google user-agent string identifies what a client claims to be; it does not prove the request came from Google. Use the string to classify the crawler, then verify the source with forward-confirmed reverse DNS or by matching the client IP against the correct Google-published CIDR list.

Google crawler and fetcher reference

Google currently groups its clients into three categories: common crawlers, special-case crawlers, and user-triggered fetchers. These groups have different purposes, robots.txt behavior, reverse-DNS patterns and IP lists.^[1][2]

Useful user-agent tokens

The following tokens are useful when filtering access logs, WAF events or crawl reports. They are not authentication:

Google can update exact user-agent strings and technical details. Use the official crawler overview as the current reference rather than copying a browser-version string permanently into firewall logic.

How to verify Googlebot correctly

Method 1: reverse DNS, then forward DNS

1. Read the actual client IP from the origin or trusted CDN log.
2. Run a reverse lookup on that IP.
3. Check the hostname against the pattern Google documents for that category. A lookalike such as googlebot.com.attacker.example is not valid.
4. Resolve the returned hostname forward.
5. Confirm that the original client IP appears among the returned IPv4 or IPv6 addresses.

# Replace with the client IP from your trusted log
IP="66.249.66.1"

# Reverse DNS
host "$IP"
# or
dig -x "$IP" +short

# Forward DNS: use the hostname returned above
HOSTNAME="crawl-66-249-66-1.googlebot.com"
host "$HOSTNAME"
# or
dig "$HOSTNAME" A +short
dig "$HOSTNAME" AAAA +short

The forward result must include the same original IP. Reverse DNS alone is not sufficient. Google’s current common-crawler patterns include hostnames under googlebot.com and geo.googlebot.com; special and user-triggered categories use different documented patterns.^[2]

Method 2: match the IP against Google’s published ranges

For automation, download the list for the relevant category and perform real IPv4/IPv6 CIDR membership checks. Searching the JSON for an exact IP string is wrong because the files contain network prefixes, not every individual address.

import ipaddress
import json
from urllib.request import urlopen

url = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json"
data = json.load(urlopen(url))
client_ip = ipaddress.ip_address("66.249.66.1")

verified = any(
    client_ip in ipaddress.ip_network(item.get("ipv4Prefix") or item["ipv6Prefix"])
    for item in data["prefixes"]
)
print(verified)

In production, set a timeout, cache the last valid response, parse both address families, refresh periodically and fail safely if retrieval or parsing fails. Do not silently replace a valid cached list with an empty or malformed response.

Official Google crawler IP lists

• Common crawlers such as Googlebot
• Special-case crawlers such as AdsBot
• User-triggered fetchers running from Google Cloud infrastructure
• User-triggered fetchers controlled by Google
• User-triggered agents reference

These files change. Store the source URL, retrieval time and upstream creationTime with your local copy so an incident can be audited against the version used at that moment.

A practical server-log workflow

1. Preserve evidence: timestamp, client IP, host, method, path, status, bytes, user-agent and request ID.
2. Identify the trusted client-IP field: behind Cloudflare, a load balancer or another reverse proxy, the origin socket may show the proxy. Trust forwarded headers only from proxies you control and have configured.
3. Classify the claimed client: map the user-agent token to common, special-case or user-triggered.
4. Verify independently: use DNS for a one-off check or CIDR matching at scale.
5. Evaluate behavior: rate, path pattern, status codes, robots.txt access and cache behavior. A verified Google request can still expose a server problem.
6. Separate crawl from index diagnosis: a 200 response in a Googlebot log does not prove indexing. Check canonical, robots directives, renderability and Search Console’s URL Inspection.

User agents and robots.txt behavior

Use the documented robots.txt token—not a full browser-like string—when you need a crawler-specific group:

User-agent: Googlebot-Image
Disallow: /private-image-path/

Common automatic crawlers respect robots.txt. Special-case and user-triggered clients can behave differently because the product or user initiated the fetch. Read the relevant crawler documentation before assuming a global User-agent: * rule controls every Google request.^[1]

Robots.txt controls crawling, not guaranteed removal from search. For indexing decisions, use the noindex, canonical and robots.txt decision guide.

Common Googlebot verification mistakes

• Allowlisting by user-agent: any script can copy the text.
• Checking only PTR: a reverse record must resolve forward to the original address.
• Using the wrong range list: an AdsBot or user-triggered request is not classified with the common-crawler list.
• Comparing exact strings instead of CIDR: the published prefixes represent networks.
• Reading the proxy IP: validate the client-IP chain before running DNS checks.
• Hard-coding IPs forever: fetch and validate refreshed lists on a controlled schedule.
• Equating crawl with indexation: a request says nothing by itself about canonical selection, noindex or ranking.
• Serving different content after verification: verification is for security and traffic handling, not for cloaking search engines.

Quick FAQ

Can Googlebot’s user agent be spoofed?

Yes. It is plain request-header text. Verify the connecting IP through Google’s documented DNS flow or IP ranges.

Should I block every unverified Googlebot request?

Not automatically. Failed verification means the identity is unverified, not necessarily that the request is malicious. Apply your normal security controls using rate, reputation, paths and behavior.

Does a verified Googlebot visit mean the page is indexed?

No. The response may redirect, fail, contain noindex, canonicalize elsewhere or be excluded for another reason. Inspect the URL in Search Console and audit the live response.

Which Google IP list should I use?

Choose the list matching the documented crawler class: common, special-case, or the relevant user-triggered category. Do not merge them blindly if the distinction matters to your policy.