About this tool

Paste the text of a robots.txt, choose a crawler such as Googlebot, and list the URLs or paths to test. Each one is reported as allowed or blocked, with the rule and line that decided it, and the file's syntax problems and unsupported directives are listed.

Matching follows the published rules (RFC 9309 as Google documents it): the most specific group for the crawler, the longest matching rule wins, and Allow wins a tie. A crawler's own implementation may differ in edge cases. The text is processed in your browser; nothing is fetched.

This page works on text you paste or load from a file. Fetching the live robots.txt of a site is an extension feature.