Check if AI search can reach your website
The ReadRankAI AI Crawler Checker reads your website's robots.txt and homepage and shows, for each AI crawler, whether it is allowed or blocked. If a search crawler like OAI-SearchBot is blocked, ChatGPT won't show your pages in its search answers (at most as a plain navigational link), no matter how good your content is. The check is free, takes a few seconds and needs no account.
How to use the AI Crawler Checker
- Enter your website, either your domain (example.com) or the full address of one page.
- Click Check crawlers. The checker reads your robots.txt and homepage.
- Read your result: a one-line verdict, a table of every AI crawler with its status, and the exact robots.txt lines to fix anything that's blocked.
Which AI crawlers it checks
Each AI company uses separate crawlers for separate jobs. The checker tests each one against your robots.txt rules.
| Crawler | Company | What it does |
|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers |
| GPTBot | OpenAI | Collects content for training OpenAI's models |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks it to |
| Claude-SearchBot | Anthropic | Crawls pages to improve Claude's search results |
| ClaudeBot | Anthropic | Collects content for training Anthropic's models |
| Claude-User | Anthropic | Fetches a page when a Claude user asks a question |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity search results |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks a question |
| Google-Extended | Google | Controls use of your content for Gemini training and grounding |
| Googlebot | Google | Crawls for Google Search, including AI Overviews |
| Bingbot | Microsoft | Crawls for Bing search and Copilot |
Sources: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers.
Search crawlers vs training crawlers
This is the difference most sites get wrong. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) affect whether your pages can appear and be linked in AI answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended) only decide whether your content is used to train AI models.
These settings are independent. You can block GPTBot to opt out of training and still appear in ChatGPT search, as long as OAI-SearchBot is allowed. Blocking Google-Extended doesn't remove you from Google Search or AI Overviews. That's why the checker counts a blocked training crawler as your choice, not a problem, and flags only blocked search crawlers.
How to read your result
- Allowed: the crawler can reach your homepage and the page you entered.
- Blocked: your robots.txt stops the crawler on both.
- Partly blocked: it can reach one but not the other.
- Couldn't read robots.txt: your server returned an error or refused the check, so the result isn't guessed.
The checker also shows whether your homepage has a noindex or nosnippet rule (both keep pages out of Google's AI Overviews), whether your robots.txt lists a sitemap, and whether you have an llms.txt file.
How to fix a blocked AI crawler
For every problem, the checker gives you the exact robots.txt lines to add. For example, to let ChatGPT search reach your site while opting out of model training:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
After you update robots.txt, run the check again. OpenAI says robots.txt changes take about 24 hours to reach its search crawler. If the checker shows everything allowed but AI tools still can't reach you, check your firewall, CDN or security plugin. Some block unknown bots by default, and robots.txt can't show that.
After the crawlers: get your content cited
Being reachable is step one. To get cited, your pages also need clear answers, specific facts and a presence on the sites AI tools trust. Score any page in the GEO Optimizer, and read our guides on how to get cited by ChatGPT and how to get mentioned by AI.
Who it's for
- Website owners who want to rule out the most basic reason ChatGPT or Perplexity never cites them.
- SEO specialists and agencies running technical checks for clients.
- Developers confirming a robots.txt change or a new security setting didn't block AI search by accident.
Frequently asked questions
What is an AI crawler?
An AI crawler is a bot that an AI company uses to visit websites, either to find pages for its search answers or to collect content for training its models. Each company names its crawlers, such as OAI-SearchBot and GPTBot from OpenAI, and your robots.txt can allow or block each one.
If I block GPTBot, will I disappear from ChatGPT?
No. GPTBot is only for model training. ChatGPT search uses OAI-SearchBot. As long as OAI-SearchBot is allowed, your pages can still appear in ChatGPT search answers.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google AI Overviews are part of Google Search, which uses Googlebot, and Google says Google-Extended doesn't affect inclusion in Search.
Why does it say "Couldn't read robots.txt"?
Your server returned an error, timed out or refused the request. Many firewalls refuse unknown bots, including this checker. Open yourdomain.com/robots.txt in a browser to see the file, and check whether your firewall treats AI crawlers the same way.
Do I need an llms.txt file?
No. llms.txt is a proposed format for giving AI tools a summary of your site. The checker shows whether you have one, but it doesn't change your result. Robots.txt is what controls whether AI crawlers can reach you.
Is the AI Crawler Checker free?
Yes. You can run up to 10 checks a day with no account and no signup.
