What robots.txt does—and what it does not do
A robots.txt file gives cooperating crawlers instructions about which paths they should request. It is a crawl directive, not an access-control mechanism: a blocked URL can still be known to search engines through links, and the file does not protect private content. Use authentication or server permissions for information that must remain private.
How to use this robots.txt tester
- Copy the exact rules from the site's root robots.txt file.
- Enter a path beginning with a slash, such as
/checkout/. - Choose the relevant crawler name and read the matching directive.
- Check the live file and the affected URL in the search engine's own tools before relying on the result.
Common crawl-rule mistakes
Look for a broad Disallow rule that unintentionally covers useful pages, a typo in a path, or a sitemap declaration that points to the wrong host. Also verify that a page is not blocked when you expect crawlers to render it. Robots rules and page-level noindex instructions have different effects; a crawler generally needs to fetch a page to see its meta robots directive.
Frequently asked questions
Does a Disallow rule remove a page from Google?
Not necessarily. Disallow controls crawling, not guaranteed de-indexing. A URL may remain known from external signals; use appropriate noindex and access controls for the intended outcome.
Can I use robots.txt to hide private pages?
No. The file is public and is not security. Protect sensitive pages with authentication or server-side access controls.
Is this tester equivalent to Google's robots tool?
No. It is a lightweight local check for common directives. Confirm important cases in Search Console and Google's current documentation.