Who reads the web
Reference: ZVL-SDR-U3L
An observed reference about AI agents and crawlers. For each agent, this site sets what its vendor documents beside what our own server logs show, and points out where the two disagree. Every documented fact has a source and a date. Every observed fact has a count and a time window.
| Agent | Vendor | Purpose | Observed | Last seen | Disagreements |
|---|---|---|---|---|---|
| GPTBot | openai | training | Not observed yet | 0 | |
| OAI-SearchBot | openai | search | Not observed yet | 0 | |
| ChatGPT-User | openai | user | Not observed yet | 0 | |
| ChatGPT agent | openai | agentic | Not observed yet | 0 | |
| ClaudeBot | anthropic | training | Not observed yet | 0 | |
| Claude-SearchBot | anthropic | search | Not observed yet | 0 | |
| Claude-User | anthropic | user | Not observed yet | 0 | |
| Googlebot | search | Not observed yet | 0 | ||
| Google-Extended | training | Not identifiable | 0 | ||
| Google-Agent | agentic | Not observed yet | 0 | ||
| Gemini user fetcher (bare "Google" user agent) | user | Not observed yet | 0 | ||
| Bingbot | microsoft | search | Not observed yet | 0 | |
| Microsoft Copilot | microsoft | user | Not identifiable | 0 | |
| PerplexityBot | perplexity | search | Not observed yet | 0 | |
| Perplexity-User | perplexity | user | Not observed yet | 0 | |
| Applebot | apple | search | Not observed yet | 0 | |
| CCBot | commoncrawl | training | Not observed yet | 0 | |
| Meta-ExternalAgent | meta | training | Not observed yet | 0 | |
| Bytespider | bytedance | training | Not observed yet | 0 |
Machine-readable: agents.json, changes.json, status.json. Each page is also available as Markdown at the same address with .md added.