What a site scan checks
A search engine or AI assistant can only show a page it can reach, read and trust. The IndexPedia site scan checks those three things on your real pages, live, the same way a crawler would.
Instead of looking at your homepage only, the scan reads every sitemap file on your site and picks pages from each kind: products, posts, locations, profiles and so on. Problems usually live in one page type, so a sample from every type finds them much faster.
- HTTP status and redirects for each page
- Load time, word count and unique words
- Titles, meta descriptions, headings and canonical tags
- noindex tags and robots.txt rules
- Text repeated from your template across many pages
- Structured data (schema) on each page
- Access for AI crawlers such as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot
Why pages get left out
Most pages that never show up in search have one of a few causes: they are blocked, they are slow or broken, or they look too much like other pages on the same site. Search engines then treat them as low value and skip them. In Google Search Console this often appears as "Crawled - currently not indexed".
AI search adds another hurdle. Tools like ChatGPT search and Perplexity use their own crawlers. If your robots.txt or firewall blocks them, your pages cannot be quoted, however good they are.
How the health score is made
Every page gets a level: fine, warning or error. The site health score weighs pages with errors three times more than pages with warnings, so one broken template counts more than a few long titles. The report shows exactly which pages made the score, so you can check it yourself.
Key terms
- Indexing
- When a search engine stores a page so it can appear in results. A page can be crawled and still not be indexed.
- Crawl
- A visit by a search engine or AI bot that downloads your page.
- Template text
- Words repeated on many pages, such as menus and footers. When most of a page is template text, search engines see little unique value.
- Canonical tag
- A tag that tells search engines which address is the main version of a page.