Which AI Crawlers Visit Your Site? A Tour of the Tracker

This is a screen-by-screen tour of the AI crawler tracker. It shows which AI bots read your site, which of those visits are verified as real, and which people arrive from ChatGPT, Perplexity, Gemini and Copilot. By the end you'll know what every chart, column and badge means, and what to do about it.

Where in the app

Your site, then the AI tab

Plan

Free on every plan

Time it takes

About 5 minutes to set up

What you need

One snippet, the WordPress plugin, or a log file

Works on

PHP sites, WordPress, Cloudflare

Three screens, one picture

The tracker has three tabs, and each one gets its own section below.

Google Analytics can't show you any of this. Analytics runs in the visitor's browser, and bots never run that code. The tracker records each visit on your server instead, before the page is sent, so nothing slips past it. It reports the visit after your page has gone out, so neither people nor crawlers wait.

Tab 1: AI crawlers

This is the screen you land on, so we'll start at the top of it.

The KPI strip

Across the top is a row of headline numbers. The biggest is Crawler visits, which reads 184,151 in the example for example.com. It counts every visit from a known AI or search crawler in the period you're looking at.

Right under it is a per-bot list, one line and one count per crawler, so you can see who does the most reading. On most sites Googlebot and Bingbot sit near the top, with GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot below them.

AI crawler visits per day

Below the strip is a daily chart covering the last 30 days. Each day is split into two kinds of visit:

• Search and user requests come from crawlers that feed live AI answers, like OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus fetchers like ChatGPT-User that open a page because a person just asked a question.
• Training visits come from crawlers like GPTBot and ClaudeBot, which collect text that may be used to train future models.

Search and user requests decide whether you show up in AI answers today. Training visits shape what models know later.

The Crawlers table

Further down is the table where you'll spend the most time. Each row is one crawler, with these columns:

• Name. The crawler's user agent, such as GPTBot or PerplexityBot.
• Company. Who runs it: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon and others.
• Visits. How many requests it made in the period.
• Pages. How many different pages it read. A crawler with many visits but few pages keeps rereading the same ones.
• Verified badge. A green Verified badge means the visits came from addresses the company publishes for that bot. The next section explains how that's checked.
• Last seen. When it last came by. If a crawler you care about hasn't been seen in weeks, something may be blocking it.

What the Verified badge means

Anyone can send a request that says "I am GPTBot", because the name is just text. The tracker therefore checks each visit's IP address against the list the crawler's owner publishes. OpenAI, Perplexity, Google, Bing and Apple all publish these lists. For crawlers verified by host name, like Googlebot, it uses a reverse and forward DNS check, which is the method the owners recommend.

Here's what that check looks like for one visit. The address 66.249.66.1 resolves to crawl-66-249-66-1.googlebot.com and sits on Google's published list, so it's the real Googlebot. The tracker runs this check automatically on every visit, so the counts in the table are real crawlers and leave out imitations. You can check any address by hand with the free Bot checker, and our guide on telling real Googlebot from fake bots shows a real and a fake result side by side.

Tab 2: Visits from AI

The second tab is about people instead of bots.

Visits per day

The top chart has a bar for each day, showing how many people clicked a link inside an AI answer and landed on your site. These are the visitors that being crawled can eventually bring you: real people, sent by an AI tool.

The tracker finds these visits from the referrer, the page the visitor came from. When someone clicks a link in ChatGPT and lands on your menu page, it's counted here.

By AI tool

Next to the chart, a list breaks those visits down by source: ChatGPT, Perplexity, Gemini and Copilot, with the number of people each one sent you.

If ChatGPT sends most of your visits, check how ChatGPT describes your business. The AI visibility check asks ChatGPT search about your business and shows what it says.

Pages people land on

The last block lists the pages these visitors arrived on, with a count for each. For Golden Crust Bakery, a bakery in Raleigh, NC, you'd hope to see the sourdough page, the catering page and the order page here. If one of them is out of date, update it and add a clear button to call, order or visit.

Tab 3: Page funnel

The third tab joins the first two together, one page at a time.

The stages

Before a page earns a visitor from AI, an AI crawler has to read it, an AI tool has to use it to answer someone, and a person has to click through. The table shows each page as a row and each stage as a column, with a green check where the page reached that stage.

A row that stops early shows you where that page gets stuck.

Filter chips

Above the table is a row of filter chips. Clicking one narrows the list to a group of pages, so you don't have to scroll through all of them. It's the quickest way to find pages that were crawled but never reached a person.

Reading the gaps

These are the most common patterns and what they usually mean:

• If the first stage has no check, no AI crawler has read the page. Link to it from your menu or home page, make sure it's in your sitemap, and check that robots.txt isn't blocking it. A Site Scan catches blocked and orphaned pages in one go.
• If the page was crawled but nothing came after, AI tools can read it but aren't using it. That's usually a content problem: a thin page, no clear facts, or nothing that answers a real question.
• If the page was used but got no visits, it shows up in answers but people don't click. A clearer title and description often help.

Every feature at a glance

Setting it up

You don't need a developer. Add your site to a free IndexPedia account, then paste one small snippet into your site. There's a version for PHP sites, a WordPress plugin, and a Cloudflare Worker if your site runs behind Cloudflare. The snippet runs on your server, so it sees crawlers that never run JavaScript.

To see the past as well, upload an access log and the tracker fills in history from the file. For a one-time look without installing anything, the free Log Analyzer reads a log on its own. Our walkthrough on how to read your server logs for AI bots goes through that report section by section.

What this means for Golden Crust

The crawler reference

These are the crawlers you'll meet most often in the Crawlers table, who runs them, and what each one is for.

If you're thinking about blocking, note that blocking a training crawler like GPTBot doesn't remove you from ChatGPT search, which uses OAI-SearchBot. Most small businesses allow at least the search crawlers, because those decide whether you appear in AI answers at all.

FAQ