What the AI Crawler Radar shows
The Radar shows how often AI and search crawlers request pages, based on real traffic from sites that use IndexPedia and server logs people chose to share. It updates every hour and covers the last 30 days. It shows only totals, never page addresses, IP addresses or site names.
Three kinds of AI crawler
AI crawlers do different jobs, and that affects what you decide to allow.
- Search crawlers, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, collect pages so they can appear in AI search answers
- User agents, such as ChatGPT-User, Perplexity-User and Claude-User, open a page because a person asked an assistant about it right now
- Training crawlers, such as GPTBot and ClaudeBot, collect content that may be used to train future models
Why so many bots are fake
Because sites tend to let famous crawlers through, scrapers copy their names. The Radar checks requests against the IP ranges each company publishes, or a reverse DNS check where the company recommends one, and counts a request as fake when it uses a crawler's name from an address that company does not use.
How to add your own numbers
Upload your server log to the free Log Analyzer and tick the box to share anonymous totals, or add your site to IndexPedia and install the tracking snippet. Either way, only counts per crawler and day are added.
Key terms
- Crawler
- A program that requests web pages automatically, for a search engine or an AI tool.
- User agent
- The name a crawler sends with each request, such as GPTBot.