Enter a website. We'll read its robots.txt file and show which AI crawlers from OpenAI, Anthropic, Google, Meta, Perplexity and others can use its pages for training, search and answers, plus every other rule in the file.
This tool fetches the robots.txt file from any website you enter and checks it against every major AI crawler, the way the crawlers themselves read it. You'll see which ones are blocked from AI training, AI search and user-triggered visits, and why. Below that, it breaks down the rules for every bot and lists restrictions, sitemaps and crawl delays. Want different rules? The AI robots.txt generator writes them for you.