What is robots.txt and is it required?

Robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that tells search engine crawlers which URLs they may or may not visit. The file uses a specific syntax defined in the Robots Exclusion Protocol (formalized as RFC 9309 in 2022). It is not strictly required — without it, crawlers assume they can visit every URL — but having one is best practice for any site over 100 pages. The file does NOT prevent pages from being indexed; it only prevents crawling. Pages already in Google index but newly disallowed in robots.txt can stay indexed (with no snippet) for weeks. To remove a page from index, use noindex meta tag instead. Common uses: blocking admin URLs (/wp-admin/, /admin/), blocking duplicate content (faceted search, cart pages), blocking staging environments, and pointing to your sitemap. Maximum file size is 500KB. Most sites have a 5-50 line robots.txt; large e-commerce sites have several hundred lines. Our generator covers all major patterns and validates against the spec.

What is the difference between Disallow and noindex?

Disallow (in robots.txt) prevents crawling — search engines never request the URL. Noindex (in meta tag or HTTP header) prevents indexing — search engines may crawl the page but will not list it in results. The two solve different problems. Use Disallow when: you want to save crawl budget on URLs Googlebot does not need to visit (admin pages, internal search results, faceted filters). Use noindex when: you want pages crawlable for link discovery but not appearing in search (low-quality landing pages, archive pages, thin content). The trap: if you use Disallow on a page that has external backlinks, Google may still index the URL with no snippet (because they cannot crawl to find the noindex). To fully exclude: use noindex without Disallow. Or use both, but only after Google has crawled the noindex once. Google clarified this in 2019: "If you want to remove a URL from search results, use noindex. Disallow alone is insufficient." Our generator includes a "remove from index" mode that produces the right combination.

How does User-agent matching work?

User-agent is the first directive in each rule block. It identifies which crawler the rule applies to. Common values: User-agent: * (matches all crawlers), User-agent: Googlebot (Google web crawler), User-agent: Googlebot-Image (Google Images crawler), User-agent: Bingbot (Bing), User-agent: Slurp (Yahoo), User-agent: DuckDuckBot (DuckDuckGo), User-agent: Baiduspider (Baidu), User-agent: YandexBot (Yandex). The matching algorithm: when a crawler hits robots.txt, it picks the most specific User-agent block that matches its name. So a rule for User-agent: Googlebot wins over User-agent: *. If no specific match exists, the * block applies. AI scrapers in 2026 added new common User-agents to consider blocking or allowing: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google AI training), PerplexityBot, Bytespider (TikTok parent company). Many publishers block these to prevent training without compensation. Our generator includes 2026 AI-bot presets.

Should I include sitemap reference in robots.txt?

Yes — it is the canonical way to tell search engines where your XML sitemap lives. Add: Sitemap: https://yoursite.com/sitemap.xml as a top-level directive (not nested under any User-agent). You can include multiple Sitemap lines if you have multiple sitemaps (large sites often split by content type: sitemap-pages.xml, sitemap-blog.xml, sitemap-products.xml). The Sitemap directive is read by Google, Bing, Yahoo, Yandex, and most other crawlers. It supplements direct sitemap submission via Google Search Console (which you should still do). Always use the full absolute URL, not relative paths. The sitemap location does not have to be on the same domain — you can host it on a CDN if needed, though same-domain is recommended for canonical clarity. Best practice: list every sitemap your site uses, including image sitemaps, video sitemaps, and news sitemaps if applicable. Our generator validates that the sitemap URL you provide returns 200 OK and is valid XML before including it.

How do I test robots.txt before deploying?

Use Google Search Console Robots Testing Tool (search.google.com/search-console/robots-testing-tool — note: classic version, requires verified property). Paste your robots.txt content and a test URL. The tool reports whether Googlebot would be allowed or blocked, and which rule applies. Bing has a similar tool in Bing Webmaster Tools. For local testing without verification, our generator includes a built-in test feature: paste candidate robots.txt content plus URLs to test, see allow/disallow results per User-agent. Always test before deploying — a single wrong rule (Disallow: / instead of Disallow: /admin/) can deindex your entire site. Common pre-deployment tests: (1) Confirm root URL "/" is allowed for Googlebot. (2) Confirm /sitemap.xml is reachable (not blocked). (3) Test 5-10 representative URLs to verify rules behave as expected. (4) Confirm intentionally-blocked URLs (admin, cart) actually return blocked. After deploying, re-test in production and monitor Google Search Console > Coverage > Errors for any unexpected exclusions.

What is a wildcard and how do I use it?

Wildcards (*) match zero or more characters in path patterns. Disallow: /search/* blocks all URLs starting with /search/. Disallow: /*.pdf$ blocks all URLs ending with .pdf (the $ anchors to end-of-URL). Disallow: /*?* blocks all URLs containing a query string. Wildcards are supported by Googlebot, Bingbot, and most major crawlers, but originally were not part of the Robots Exclusion Protocol. The RFC 9309 standardization in 2022 codified them. Common patterns: Disallow: /admin/ blocks /admin/ and everything under it. Disallow: /*?utm_source=* blocks all URLs with utm_source parameter (useful for blocking tracking-tagged duplicate URLs). Disallow: /print/ blocks print-friendly versions. Avoid: overly broad wildcards like Disallow: /*-archived* which can accidentally block legitimate URLs. Always test wildcards in Google Search Console testing tool. Our generator visualizes wildcards and shows you sample matching URLs before you commit the rule.

Can robots.txt protect sensitive content?

No — robots.txt is publicly readable and not a security mechanism. Anyone can fetch yoursite.com/robots.txt and see exactly which URLs you are blocking. If you list /admin/ as disallowed, malicious actors know exactly where to look. For real security: use HTTP authentication (basic auth, OAuth) on the page, IP-based access control, or move the content behind authenticated routes that return 403 to non-logged-in visitors. Robots.txt is for politeness — well-behaved crawlers respect it; malicious crawlers ignore it. Common mistakes: listing internal API endpoints in Disallow (signals where to attack), listing staging URLs (signals to scrape staging), listing customer-data URLs (signals where to find PII). Best practice: never list sensitive paths in robots.txt. Use noindex meta tags for content that should not appear in search but does not need cryptographic security, and proper authentication for content that genuinely needs protection. Our generator warns when patterns look like they might leak sensitive paths.

How do I block AI scrapers in 2026?

In 2026, blocking AI training crawlers is increasingly common. Add specific User-agent rules: User-agent: GPTBot Disallow: / (blocks OpenAI). User-agent: ClaudeBot Disallow: / (blocks Anthropic). User-agent: CCBot Disallow: / (blocks Common Crawl, which feeds many LLMs). User-agent: Google-Extended Disallow: / (blocks Google AI training without affecting Googlebot search crawling). User-agent: PerplexityBot Disallow: /. User-agent: Bytespider Disallow: / (blocks TikTok/ByteDance training). Blocking these does not affect search rankings — Googlebot for search and Google-Extended for AI are separate User-agents. Note: PerplexityBot and others reportedly ignore robots.txt despite advertising compliance. Cloudflare, Vercel, and Akamai now offer firewall-level blocking that works regardless of bot honesty. Our generator includes a "Block AI training" preset that adds rules for the 7 major AI scraping User-agents. Many publishers (NYT, Reuters, BBC) added these blocks in 2024-2026.

What is the Crawl-delay directive?

Crawl-delay tells crawlers how many seconds to wait between requests to your site. Example: Crawl-delay: 10 means the crawler should wait 10 seconds between successive requests. Useful when your server cannot handle aggressive crawling (typically smaller hosting tiers). However: Google does NOT support Crawl-delay. Set crawl rate for Googlebot in Google Search Console > Settings > Crawl rate instead. Bing, Yandex, and Yahoo do honor Crawl-delay. Reasonable values: 5-10 for normal sites, 30+ for tiny sites on shared hosting, 1 or omit for sites on dedicated infrastructure. Higher values reduce indexing speed: a site with 10,000 URLs and Crawl-delay: 30 takes 3.5 days minimum to crawl. For most modern sites on cloud hosting (Vercel, Netlify, AWS), Crawl-delay is not needed and may slow indexing without benefit. Our generator suggests Crawl-delay only when you indicate small/shared hosting and warns it does not apply to Google.

Can I have multiple robots.txt files for subdomains?

Each subdomain has its own robots.txt. blog.yoursite.com/robots.txt is independent of yoursite.com/robots.txt. Each must be configured separately. This is useful for: (1) Different rules per subdomain (e.g., block staging.yoursite.com entirely while allowing all of yoursite.com). (2) Multi-property setups where blog and main site have different policies. (3) E-commerce sites with separate shop subdomains. Common gotcha: WWW vs non-WWW are technically different subdomains, so you need www.yoursite.com/robots.txt and yoursite.com/robots.txt to both exist (typically both serve the same content via 301 redirect). For HTTP vs HTTPS: most servers serve the same robots.txt for both protocols, but verify. Subdirectory paths (yoursite.com/blog/) share the parent domain robots.txt — there is no /blog/robots.txt. If you need different rules for /blog/, write rules referencing /blog/* paths in the main robots.txt. Our generator handles single-domain output; for multi-subdomain setups, generate one file per subdomain.

Does robots.txt affect site speed or performance?

Indirectly. Robots.txt itself is a tiny text file (typically <2KB) that loads instantly. The performance impact comes from what it blocks. Disallowing pages that Googlebot would otherwise crawl saves your crawl budget — Googlebot has a daily request limit per site, so blocking unnecessary URLs lets it focus on important ones. For a site with 100,000 URLs and 50% being thin or duplicate content, properly blocking those URLs in robots.txt can effectively double Googlebot crawl efficiency. The opposite trap: blocking essential resources (CSS, JS files) prevents Googlebot from rendering pages correctly, which can hurt rankings. Google explicitly recommends NOT blocking CSS, JS, or image directories — modern Googlebot needs to render pages like a browser to evaluate them. Disallow: /static/css/ or Disallow: /js/ is almost always wrong in 2026. Our generator validates that you are not accidentally blocking critical resources by warning about common mistakes.

How often should I update robots.txt?

Whenever your site structure changes meaningfully. Specific events that should trigger a robots.txt update: launching a new site section (add to allow rules or sitemap reference), retiring an old section (consider Disallow), adding tracking parameters (Disallow URL patterns to prevent duplicate content), launching a multi-language site (consider per-locale rules), migrating CMS (paths typically change), or adding AI training blocks (the AI bot landscape evolves quarterly). Outside these events, you do not need to touch robots.txt. Most stable sites update theirs 0-2 times per year. Our customers running e-commerce typically update every 6 months as catalog structures evolve. Always test before deploying changes — even small typos can cause large coverage shifts. After deploying changes, monitor Google Search Console > Coverage report for 2-4 weeks to catch unintended exclusions.

Free Robots.txt Generator

Generate robots.txt with allow/disallow rules per user-agent. Sitemap, crawl delay. Free, instant.

4.6on G2

4.8on Trustpilot

Used by 25,000+ marketers

Default & sitemap

Default - All Robots are:

Crawl-Delay:

Sitemap:

Search Robots:

Google

Google Image

Google Mobile

MSN Search

Yahoo

Yahoo MM

Yahoo Blogs

Ask/Teoma

GigaBlast

DMOZ Checker

Nutch

Alexa/Wayback

Baidu

Naver

MSN PicSearch

Restricted Directories:

The path is relative to root and must contain a trailing slash "/"

Your generated robots.txt

# Output updates as you change settings.

What this tool does

Robots.txt Generator delivers fast, reliable results for generate robots.txt with allow/disallow rules per user-agent. sitemap, crawl del.

Designed to fit into your existing SEO and content workflow with no setup overhead.

How to use it

Five steps.

Pick your platform preset

Choose WordPress, Shopify, Webflow, Next.js, or generic. Preset adds standard Disallow rules for that platform automatically.

Add custom rules

Append your own Allow / Disallow patterns. Use wildcards (*) for prefix matching and $ for end-of-URL anchoring.

Reference your sitemap

Add the full absolute URL of your sitemap.xml. Multiple sitemap lines if you have separate sitemaps for pages, blog, and products.

Toggle AI bot blocks

Enable the AI bot preset to block GPTBot, ClaudeBot, Google-Extended, and 4 other 2026 AI training scrapers.

Generate, test, deploy

Copy output, deploy as /robots.txt at your domain root. Test rules in Google Search Console Robots Testing Tool before relying on them.

When teams use it

Six common workflows.

Block staging environment from indexing

Generate a Disallow: / robots.txt for staging.yoursite.com to prevent Google from indexing pre-production content. Combine with HTTP basic auth for full protection. Required before any client launch.

Save crawl budget on faceted commerce URLs

E-commerce filters (size, color, price) generate millions of duplicate URLs. Use Disallow: /*?filter=* and similar wildcards to block crawling. Lifts crawl efficiency on important product pages 3-5x.

Block AI training scrapers

Add 2026-current AI bot blocks: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider. Generator includes a one-click preset for the 7 major AI training User-agents.

Reference sitemap for multi-sitemap sites

Add multiple Sitemap directives to point search engines at sitemap-pages.xml, sitemap-blog.xml, sitemap-products.xml. Improves index discovery for sites over 50,000 URLs.

Configure platform-specific rules (WordPress, Shopify)

Each major CMS has standard paths to block (/wp-admin/, /cart/, /checkout/). Generator includes platform presets that capture the right Disallow rules without you needing to remember each.

Block aggressive crawlers harming server load

Use Crawl-delay or full Disallow for User-agents that hammer your site (SemrushBot, AhrefsBot, etc. — disable for SEO research at your discretion). Generator includes detection of high-volume crawlers.

Platform guides

Integrate with major platforms.

WordPress

Generate the robots.txt content with WordPress mode (includes /wp-admin/ Disallow).
Upload via FTP to your site root, or use a plugin like Yoast SEO > Tools > File Editor.
Verify yoursite.com/robots.txt loads correctly.
Test rules in Google Search Console Robots Testing Tool.
Submit your sitemap to Google Search Console.

Shopify

Shopify generates robots.txt automatically. As of 2021, Shopify Plus stores can customize it.
In Shopify admin (Plus only), navigate to Online Store > Themes > Edit code > robots.txt.liquid.
Generate custom additions from our tool and inject into the Liquid template.
Save and verify yoursite.com/robots.txt.
Test specific URLs in Google Search Console.

Next.js / Vercel

Generate robots.txt content from our tool.
Save as public/robots.txt in your Next.js project.
For dynamic robots.txt, use app/robots.ts (Next.js 13+) or pages/robots.txt.js with getServerSideProps.
Deploy via vercel --prod.
Verify at yoursite.com/robots.txt and test in Google Search Console.

Webflow

Generate robots.txt content.
In Webflow Designer, navigate to Project Settings > SEO.
Paste content into the Robots.txt field.
Publish the site.
Verify yoursite.com/robots.txt and test rules in Google Search Console.

Static / Apache

Generate robots.txt content.
Save as a file named robots.txt (case-sensitive) in your web root directory.
Set file permissions to 644 (chmod 644 robots.txt).
Verify yoursite.com/robots.txt loads via curl or browser.
Test rules in Google Search Console Robots Testing Tool.

Grigora vs. alternatives

Side-by-side.

Capability	Grigora	Yoast SEO	SEOptimer	Merkle	Manual
AI bot presets (GPTBot, ClaudeBot, etc.)	Yes	No	Limited	Yes	Manual
Wildcard pattern testing	Yes	Yes	No	Yes	Manual
Multi-sitemap reference	Yes	Yes	Yes	Yes	Manual
Platform presets (WordPress, Shopify, etc.)	Yes	No	Limited	No	Manual
Validates against RFC 9309	Yes	No	No	Limited	No
Free without signup	Yes	Trial only	No	Plan-capped	Yes
Built-in URL test against generated rules	Yes	No	No	Yes	No
Sensitive-path warning	Yes	No	No	No	No

Common errors and fixes

Eight issues users hit.

Disallow: / accidentally blocked entire site

Cause: Single character typo: extra slash treated as block-everything rule.

Fix: Replace with the specific path you intended (Disallow: /admin/) or remove the rule. Re-deploy and verify in Google Search Console Robots Testing Tool that root "/" is allowed.

Pages still appearing in index after Disallow

Cause: Disallow prevents crawling but does not deindex pages that already exist in Google index.

Fix: For removal, add noindex meta tag to the page and temporarily remove the Disallow so Google can crawl and see the noindex. Re-add Disallow after Google processes the noindex.

CSS or JS blocked, harming rankings

Cause: Old robots.txt blocked /static/, /assets/, or similar resource directories.

Fix: Remove Disallow rules for CSS, JS, and image directories. Modern Googlebot needs to render pages fully. Test deployment in Google Search Console Mobile-Friendly Test to confirm rendering works.

Sitemap directive ignored by crawlers

Cause: Sitemap URL was relative or inaccessible (404 or behind auth).

Fix: Use full absolute URL: Sitemap: https://yoursite.com/sitemap.xml. Verify the URL returns 200 OK in incognito browser. Submit also via Google Search Console Sitemaps section.

Wildcard rule matched more URLs than intended

Cause: Wildcard placed too broadly (e.g., Disallow: /*-test* matched legitimate URLs).

Fix: Test the wildcard in Google Search Console Robots Testing Tool against 10+ representative URLs. Tighten the pattern to be more specific. Use $ anchor for "ends-with" matching.

Crawl-delay not respected by Googlebot

Cause: Google does not support the Crawl-delay directive.

Fix: Set crawl rate for Google in Google Search Console > Settings > Crawl rate. Crawl-delay still works for Bing, Yandex, Yahoo, but is ignored by Google as of 2019.

AI bot rules ignored

Cause: Some AI scrapers (PerplexityBot, others) reportedly ignore robots.txt despite stating compliance.

Fix: Add firewall-level blocking via Cloudflare WAF, Vercel Firewall, or AWS WAF using User-agent header rules. Combine with robots.txt for compliant bots and firewall for non-compliant ones.

Robots.txt 403 or 500 from CDN

Cause: CDN cache returned an error, causing Google to assume default (allow all) or skip site temporarily.

Fix: Verify robots.txt loads in incognito. Check CDN logs for 4xx/5xx on the path. Fix at origin server, then purge CDN cache for /robots.txt to propagate fix.

Original data

2026 study.

500KB

Maximum file size for robots.txt

3-5x

Crawl-budget lift from blocking faceted URLs

2026 AI scrapers commonly blocked

0-2 times

Sites updating robots.txt yearly

Frequently asked questions

Twelve answers.

Related free tools

Other utilities.

Try Robots.txt Generator now

Free, unlimited, no signup.

Try the Tool