Robots.txt is a text file at the root of a website that tells crawlers which pages they should or shouldn't crawl. It's a crawl directive, not a security measure.

Does blocking a URL in robots.txt prevent it from ranking?

Blocking prevents crawling, not necessarily indexing. Google can still index a URL it hasn't crawled if other pages link to it. To prevent indexing, use a noindex meta tag instead.

Robots.txt Tester

Fetch any site's robots.txt and test whether a specific URL path is blocked for a given crawler.

Enter a site URL above to load its robots.txt

How it works

Start getting insights in minutes

Test any URL against your robots.txt rules instantly.

🌐

Enter your domain and URL to test

Provide your domain (so the tool can fetch /robots.txt) and the specific URL path you want to test against the rules.

🤖

Choose the user-agent

Select the crawler you want to test for - Googlebot, Bingbot, GPTBot, or a specific bot. Rules can differ per user-agent.

✅

See the result and matching rule

The tool shows whether the URL is allowed or disallowed, which specific rule matched, and displays the full parsed robots.txt for context.

Why use it

Everything you need from Robots.txt Tester

Test any URL against your robots.txt rules instantly.

🛡️

Confirm important pages aren't blocked

Accidentally blocking key pages in robots.txt is more common than you'd think - especially after CMS upgrades or template changes. This tool verifies they're crawlable.

🤖

Test for specific bots, not just Googlebot

With AI crawlers like GPTBot and CommonCrawl now significant, you may want to verify rules apply correctly to non-Google bots as well.

🔍

See which rule is matching

When a URL is blocked, the tool shows exactly which line in the robots.txt is causing it - so you know which rule to update rather than guessing.

📋

Parse the full file, not just test one URL

The full robots.txt is displayed and parsed so you can see all rules across all user-agents at once - useful for auditing a file you haven't looked at in a while.

⚠️

Spot conflicting rules

Robots.txt files sometimes have allow and disallow rules that conflict. The tester highlights ambiguous rule interactions that might produce unexpected crawl behaviour.

FAQ

Common questions

It fetches a site's robots.txt file and tests whether specific URLs are blocked or allowed for a given user-agent, showing which rule is responsible for the result.

Robots.txt Tester

All your SEO answers, completely free.

No account needed for standalone tools. Google OAuth for GSC tools - read-only, no data stored.

Read-only OAuth · no payment · no data reselling