Robots.txt Tester
Test any URL and user-agent against robots.txt rules. Parses allow, disallow, and crawl-delay directives.
ALLOWED
Matched agent: Googlebot
Matched rule: Allow: /
Crawl-delay: 0.5s
Parsed rules (2)
User-agent: *
Disallow: /admin/
Disallow: /private/
Crawl-delay: 1
User-agent: Googlebot
Allow: /
Crawl-delay: 0.5
How robots.txt works
The robots.txt file sits at the root of your domain and tells crawlers which pages they may or may not access (per the Robots Exclusion Protocol, RFC 9309). User-agent blocks are matched by substring. The most specific Allow/Disallow rule (by path length) takes precedence. A missing or empty robots.txt means all crawlers may access all pages.Crawl-delay asks bots to wait N seconds between requests.
Private & free — this tool runs entirely in your browser.
Recommended: Namecheap — Register a domain for your next project — from $1.98/yr.affiliate
Related Web & SEO tools
Meta Tag Generator
Build SEO title, description, and viewport meta tags.
Open Graph Generator
Generate Open Graph and Twitter Card meta tags.
Robots.txt Generator
Build a robots.txt file with per-bot rules. Block AI crawlers, set crawl delays, define sitemaps.
UTM Campaign URL Builder
Append UTM parameters to build trackable campaign URLs.
Hreflang Tag Generator
Generate hreflang link tags for multilingual SEO.
Sitemap XML Generator
Build an XML sitemap from a list of URLs.
SERP Snippet Preview
Preview how your page title and meta description appear in Google search.
Meta Tag Analyzer
Paste a page's HTML head to audit its title, meta, and social tags.