Robots.txt Tester

See if robots.txt allows crawling.

Website URL

Rules correct? Next is the site itself

A crawlable site is only worth as much as the pages behind it.

Grigora builds the site, the newsletter and the store on one platform, with the robots.txt, sitemap and canonical tags generated from your actual pages instead of maintained by hand in a text file nobody remembers editing.

Start free Explore the builder

How to test a robots.txt file

Four steps from file to verdict.

Enter your domain

The live robots.txt for that origin is fetched for you. Or switch to the paste tab and test a draft you have not published yet.

List the URLs to test

One per line. Full URLs or bare paths both work, including query strings.

Pick the crawler

Choose Googlebot, Bingbot or any other name. The tool selects the group that crawler would use.

Read the matching rule

Each URL shows allowed or blocked plus the exact line that decided it, so you know what to edit.

How to test a robots.txt file, step by step

Real screenshots of the current tool, using sample inputs. Long results are shown as a preview.

1
Enter a domain and the URLs to test
The live robots.txt for that origin is fetched for you. List one URL per line, as full URLs or as bare paths.
2
Pick the crawler to test as
A group that names Googlebot overrides the star group entirely, which is the part that catches most people out.
3
Review the result
Select Test robots.txt to run the tool. This screenshot shows the result returned for the sample input.

When this check matters

Six moments a robots.txt test saves real traffic.

After a staging launch

Staging files carry "Disallow: /". Confirm it did not ship to production with the code.

When pages stop being indexed

Check whether a rule is blocking the crawl before you rewrite the page.

Before editing a live file

Test the draft here first. A bad rule can remove a section from search within days.

Debugging faceted URLs

See whether a wildcard rule catches your sort and filter parameters, and only those.

Checking one crawler at a time

A specific Googlebot group overrides the * group entirely, which surprises most people.

Auditing an inherited site

Read what the previous team actually blocked, line by line, before you change anything.

Finding robots.txt on your platform

Where the real file lives, by platform. Always test the served output rather than the template.

Shopify

Open Online Store, then Themes.
Edit code and open robots.txt.liquid.
Copy the rendered output from yourstore.com/robots.txt and paste it above.

WordPress

Check whether an SEO plugin generates the file virtually.
Open yoursite.com/robots.txt in a browser.
Copy the whole response and test it here.

Next.js

Look for public/robots.txt or an app/robots.ts route.
Build locally and request /robots.txt.
Paste the served text, not the source template.

Grigora

Open Site settings, then SEO.
Review the generated robots.txt.
Test it here before pointing the domain live.

Grigora vs. other robots.txt testers

Side by side.

Capability	Grigora	Online checkers	Search Console	Crawler suites
Runs without fetching your site	Yes, paste a draft	Needs a live URL	Needs verified site	Needs a live URL
Quotes the matching line	Yes, with line number	Sometimes	Yes	No
Bulk URL testing	Unlimited list	One at a time	One at a time	Varies
Flags syntax problems	Yes, with the reason	Basic	Yes	No
Signup required	No	No	Yes	Yes

Common errors and how to fix them

Eight robots.txt mistakes that quietly cost traffic.

Disallow: / left in production

Cause: The staging robots.txt was deployed with the build.

Fix: Remove the line. Recrawling takes days, so check this the hour a site goes live.

Blocking a page to deindex it

Cause: Disallow used where noindex was meant.

Fix: Blocking stops the crawl, so the noindex tag is never read and the URL can stay listed. Allow the crawl and serve noindex.

Blocking CSS and JavaScript

Cause: Disallow: /assets/ or /wp-includes/.

Fix: Google renders pages. Blocked CSS and JS make the page look broken to the renderer, which affects how it is assessed.

Rule under the wrong group

Cause: A blank line split the group in two.

Fix: A user-agent line starts a group and rules belong to the group above them. Keep each group contiguous.

Expecting * to also apply

Cause: A Googlebot group exists, so the * group is ignored by Googlebot.

Fix: Only one group applies per crawler. Repeat any shared rules inside the specific group.

Case mismatch in paths

Cause: Disallow: /Admin/ against a /admin/ URL.

Fix: Paths are case sensitive. Directive names are not, but everything after the colon is.

Relative sitemap URL

Cause: Sitemap: /sitemap.xml.

Fix: The value must be absolute, with the scheme and host. A relative value is discarded.

Missing $ on an extension rule

Cause: Disallow: /*.pdf blocks /file.pdf?ref=x and also /pdf-guide/page.

Fix: Add $ to anchor the end, and remember it then stops matching URLs with query strings.

Limits worth memorising

The four numbers that explain most surprising results.

500

Kilobytes of robots.txt Google reads before it stops parsing

Groups that apply to any single crawler, never two

Wildcard characters supported in paths: * and $

404

Status on robots.txt that means every URL is allowed, not an error

Frequently asked questions

Ten answers about robots.txt behaviour.

It applies the same matching rules a crawler applies, then tells you whether a given URL is allowed or blocked and which line produced that result.

When more than one rule matches a URL, the rule with the longest path pattern wins, regardless of whether it is Allow or Disallow and regardless of its position in the file.

A * matches any sequence of characters, so /*?sort= matches any URL containing ?sort= after any prefix. A $ at the very end anchors the match to the end of the URL, so /*.pdf$ matches /a.pdf but not /a.pdf?ref=1.

Only one group applies to any crawler, the most specific one that names it. Googlebot reads the Googlebot group and completely ignores the * group, so any shared rule has to be repeated inside the specific group.

No. Blocking prevents crawling, not indexing. A blocked URL that other pages link to can still appear in results, usually without a description.

Directive names are not, so Disallow and disallow behave identically. Path values are case sensitive, so Disallow: /Admin/ does not block /admin/.

A line reading "Disallow:" with nothing after the colon imposes no restriction at all. It is the conventional way to write a group that permits everything, and it is not the same as "Disallow: /".

No. Googlebot ignores the directive entirely. Bing and Yandex do read it. If you need to slow Googlebot down, adjust the crawl rate in Search Console rather than adding a line here.

At the root of each host and scheme, so https://example.com/robots.txt covers only that exact origin. A subdomain such as shop.example.com needs its own file, and a file at any other path is ignored.

Yes, and that is the safer order. Switch to the paste tab and nothing is requested from your server at all, so you can test a draft against your URLs and only then deploy.

Related free tools

Other utilities that pair with the Robots.txt Tester.

Build a site that is crawlable by default

Pages, newsletter and store in one place, with the technical SEO handled for you. Try it free for 14 days, no card needed.

Start building free