Why track AI crawlers
Analytics tools do not show bots, so most site owners have no idea whether AI tools read their pages at all. The AI crawler tracker records every visit from known AI and search crawlers: which bot, which page, when, and what your server answered.
A page that AI search crawlers never visit cannot appear in AI answers, so spotting those pages early tells you where to look.
Verified, not just named
Anyone can send a request that says "GPTBot". IndexPedia checks each visit against the IP addresses the bot's owner publishes, such as OpenAI, Perplexity, Google and Bing, so imitations are left out of the numbers.
- Visits per bot, per day
- Pages AI crawlers read most, and pages they never reach
- Errors and blocks crawlers ran into
- Visitors sent to you by ChatGPT, Perplexity, Gemini and Copilot
How it is installed
Add one small snippet to your site, or upload your server log to see the past. The snippet runs on your server, so it sees crawlers that never run JavaScript, and it reports each visit only after your page has been sent.
Key terms
- ChatGPT-User
- OpenAI's agent that opens a page when a ChatGPT user asks about it.
- OAI-SearchBot
- OpenAI's crawler for ChatGPT search results.
- AI referral
- A visitor who arrives by clicking a link in an AI answer.