Free tool
AI Crawler Checker: ChatGPT, Claude and Perplexity
Enter your site address. We check your robots.txt rules and the real response at the door for each AI and search crawler. Free; just sign in with Google.
Do the crawlers actually come?
Even when the rules allow it, only your own server knows whether a crawler actually visits. Our WordPress plugin counts AI crawlers and visitors arriving from ChatGPT; BoostMaster also tracks your brand in ChatGPT and Gemini answers every day.
Frequently asked questions
Which crawlers should I block?
If you want to be cited in answers, don't block search and user crawlers (such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot). If you don't want your content used for model training, you can block only the training crawlers (such as GPTBot, ClaudeBot, Google-Extended).
Does Cloudflare block AI crawlers?
If Cloudflare's "AI Crawl Control" or "Block AI bots" settings are on, crawlers may be stopped at the door without ever reading your robots.txt. This tool shows that as "stopped at the door".
Do I need llms.txt?
No. Some AI tools read it but Google does not use it. Having one does no harm; not having one is not a gap.