AI Ranked Robots.txt AI Crawler Checker

Use Dang.ai’s AI Crawler robots.txt checker to see how your robots.txt affects major AI crawlers. Paste your full robots.txt, then test specific paths (for example, “/”, a pricing page, or a docs page). The results show whether each AI user agent is Allowed or Blocked for the path you enter.

The checker compares your rules against a list of AI-related user agents, including training crawlers and AI search bots. For each agent, you get the exact rule line that decided the outcome, so you can quickly find what in your file is blocking (or allowing) a particular crawler. It also uses group logic based on User-agent lines: bots apply the group that names them, and fall back to * only when nothing explicitly matches.

This tool is intended for site owners and teams working on AI search visibility—especially publishers and SaaS companies that want to control which AI systems can fetch their content. If you’re blocking some bots on purpose (for licensing, privacy, or data protection reasons), the bot-by-bot breakdown helps you verify your policy.

There’s no pricing or subscription mentioned for this free tool. It runs in your browser, and it can’t fetch your robots.txt automatically due to browser restrictions—copy your robots.txt contents directly into the input.

Why can’t the checker fetch my robots.txt automatically?

Your browser blocks pages from reading other sites’ files (CORS). Paste your robots.txt content into the tool input instead, and keep it located at the root path (https://yourdomain.com/robots.txt) for correct testing.

What paths should I test, if I only test “/”?

Test more than the homepage. Use the path box for key pages you care about being crawled and potentially cited—like /pricing/, your top blog entry, and a docs page—since old or overly broad rules often block specific sections.

If I block GPTBot, will my site disappear from ChatGPT search?

Blocking training crawlers controls what models may learn from future crawls, not whether a site can appear in AI search results. For outcomes in ChatGPT search, focus on the AI search agents (the checker helps you see those separately from training bots).

Do AI crawlers always obey robots.txt?

Not all AI-related user agents behave the same way. Some may ignore robots.txt for certain fetch behaviors, while others do respect it—use the checker’s Allowed/Blocked results as a starting point and verify against your access logs if needed.

If I add a bot-specific group, does it still follow my wildcard (*) rules?

No. When you create a group that names a bot explicitly, that bot uses only that group’s rules. This can override what would otherwise be covered by your * group, so the checker’s recommended configuration copies * rules into named groups to avoid accidental blocks.

How quickly do robots.txt changes take effect?

The checker can show the updated behavior immediately for the rules you paste in, but real crawler behavior depends on when crawlers re-fetch your robots.txt and how your hosting/CDN serves it. If a bot still seems blocked after changes, check your access logs and any bot-management settings.