AI search visibility
Search is turning into answers. When someone asks ChatGPT or Perplexity or Google for a recommendation, a handful of sites get read and cited and everything else is invisible. This is where the free tools and the open data for that live.
Check a site
- AI visibility checker
Paste a website address and see whether the crawlers behind ChatGPT, Perplexity and Google's AI answers are allowed to fetch it, and whether the page sends real text or an empty JavaScript shell.
- robots.txt user-agent checker
Finds user-agent lines that look fine but match nothing, because the robots.txt standard does not allow digits or dots inside a crawler name.
- llms.txt generator and validator
Builds an llms.txt from the pages a site actually has, or checks an existing one against the spec.
Open data
- AI crawler census
An open measurement of the top 5,000 sites: which AI crawlers each one allows or blocks, how many ship an llms.txt, and how many homepages send real text without JavaScript. Published as JSON under CC BY 4.0.
- Every site in the census that blocks AI search
The full named list, with what each one blocks, linking through to that site's own live result.
- Who blocks AI search: tracked board
A smaller panel of recognisable sites, re-checked over time, so a site changing its mind shows up with a date.
Reference
- AI crawler reference
What each crawler is for, who runs it, the exact robots.txt token it obeys, and whether blocking it costs you citations or only training.
- Guides
Plain answers to the questions people actually search: why a site is not showing up in ChatGPT, whether to block GPTBot, how Google's AI Overviews pick sources.
How this is measured
Every result comes from fetching the site's own public files: robots.txt, the homepage as a crawler receives it, the sitemap, llms.txt. Nothing is guessed and nothing you type is stored.
Results are three-state. Allowed, blocked, and unknown when a file cannot be fetched. A checker that turns "I could not tell" into "you are fine" is worse than no checker, so this one says unknown and means it.
Robots.txt is read the way the standard says a crawler reads it: the most specific matching group wins, and a catch-all group applies when nothing names the crawler. Counting only the sites that name a crawler and disallow everything undercounts real blocks.
The census is published as JSON under CC BY 4.0, one row per site, so anyone can check the numbers instead of taking them on trust.