Robots.txt Directive & Syntax Tester
Validate search crawler permissions, test User-Agent directives (Googlebot, Bingbot, GPTBot), and verify XML sitemap declarations.
Robots.txt Best Practices
Ensure your robots.txt file is placed in the root directory (e.g., https://example.com/robots.txt). Never disallow CSS or JavaScript assets (such as /_next/ or /static/), as Google and modern search engines require these to properly render JavaScript and evaluate Core Web Vitals.
What is the purpose of robots.txt?
Robots.txt tells web crawlers (like Googlebot and Bingbot) which URLs they can or cannot request from your site. It is primarily used to manage crawl budget and prevent crawling of admin or search filter pages.
Does robots.txt prevent a URL from appearing in Google search?
No! If other sites link to your page, Google may still index the URL without crawling its content. To ensure a page is not indexed, use the `noindex` meta tag rather than blocking it in robots.txt.