AI Crawlers & Bot Verification
How to tell a real GPTBot from a scraper spoofing its User-Agent: HTTP Message Signatures and AI crawler analytics
AI agent verification checks the cryptographic signature of an HTTP request (HTTP Message Signatures) that genuine OpenAI, Anthropic, Perplexity and Google crawlers attach to their requests. A spoofed User-Agent without a valid signature is marked unverified.
Why verify AI crawlers
User-Agent is trivial to spoof: any scraper can call itself GPTBot and bypass your robots.txt rules. Genuine AI crawlers sign requests per the HTTP Message Signatures standard — the public key lives at /.well-known/http-message-signatures-directory. Signature verification lets you reliably distinguish an official agent from an imitator and apply different decisions: full access for a verified bot, throttling or blocking for a scraper.
What GEO Engine does
- Accepts crawl events from the site (CMS agent or JS snippet) together with signature headers
- Verifies the signature: keyid → provider directory → cryptographic check (Ed25519/RSA), validity window and replay protection
- Stores the verification result: verified / failed + reason (signature_mismatch, expired, unknown_key, etc.)
- Shows the share of verified hits per AI agent in analytics — separately from UA-only traffic
- A public /bot-auth/verify endpoint lets an agent verify its own signature
Important limitation: a User-Agent string alone is not proof — only a valid signature is. Traffic without signature headers is honestly counted as unverified even if the User-Agent looks like GPTBot.
How does a verified AI bot differ from a regular crawler in the stats?
A verified bot passed HTTP Message Signature verification — it is a proven official provider agent. Regular traffic with a similar User-Agent stays unverified and may be a scraper.
Do I need to configure anything on my site for verification?
Just pass request headers in crawl events — CMS agents and snippets do this automatically. Providers publish their keys in a well-known directory that we check during verification.