AI Crawler Checker
This tool checks your robots.txt and HTTP headers to show whether AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot can reach your site, and whether any blocks would remove you from AI answers. Enter a domain to see a per-bot readout grouped by retrieval and training.
What is robots.txt?
robots.txt is a plain-text file at the root of your domain that tells well-behaved crawlers which URLs they may fetch. It uses simple User-agent and Allow / Disallow directives. Most major AI crawlers respect it — but a few don't, which is why this tool flags them separately.
Which AI crawlers matter for visibility?
For citations in AI answers, the retrieval crawlers matter most: OAI-SearchBot (ChatGPT search), PerplexityBot, ClaudeBot, and Google's AI Overviews crawlers. Training crawlers like GPTBot and Google-Extended only affect future model training.
Why blocking GPTBot doesn't hide you from ChatGPT
ChatGPT's live "search the web" feature uses OAI-SearchBot, a separate user-agent. Many sites block GPTBot to keep content out of training data, but leave OAI-SearchBot allowed so ChatGPT can still find and cite their pages. Both are legitimate choices — just make sure you know which one you're making.
What to do if AI crawlers are blocked
Edit robots.txt at your domain root and add explicit Allow rules for the retrieval bots that matter. Remove blanket Disallow: / entries that catch every crawler. If a CDN or WAF is rate-limiting AI bots, allowlist their IP ranges. Re-run this check after changes propagate.