How does Google detect a noindex directive?

Google scans for the noindex value in two places: the meta robots tag in the HTML head ( ) and the X-Robots-Tag HTTP response header. Both are honored equally; if either says noindex, Google removes the page from its search index. Critically, Google must crawl the page to see the directive, which means a robots.txt block can prevent Google from ever discovering the noindex tag, leaving the page indexed indefinitely. The Noindex Checker scans both the meta tag and HTTP headers, plus checks robots.txt for any conflict that would block the crawler from reading the directive. About 19% of attempted noindex implementations in our 2026 sample had a robots.txt blocking conflict that defeated the noindex; identifying this combination is one of the most valuable findings the tool produces.

What is the difference between noindex, nofollow, and disallow?

Noindex prevents the page from appearing in search results; nofollow prevents PageRank from flowing through links on the page; disallow (in robots.txt) prevents crawlers from accessing the page at all. They serve different purposes and are sometimes confused. Disallow is the bluntest: it blocks crawl, but Google can still index the URL string from external links and show "no information available" in results. Noindex requires the page to be crawlable so Google can read the directive. Combining disallow + noindex breaks noindex because Google never sees the directive. The recommended pattern: use noindex (allow crawling) for pages you want excluded from search but might link to externally; use disallow only when you want to prevent crawl entirely (typically server load or staging sites). The Noindex Checker validates these combinations and warns about conflicts.

How long does it take for noindex to remove a page from Google?

Typically 1 to 14 days from the time Google recrawls the page. For high-authority pages with frequent crawl frequency (typically 1 to 3 days between crawls), removal happens within a week. Low-authority pages with monthly crawl frequency can take 30+ days. To accelerate, submit the URL via Search Console URL Inspection > Request Indexing; this triggers an immediate recrawl and noindex detection, typically removing the page within 24 to 72 hours. The Noindex Checker reports the meta robots state on the live page; if your tag is present and Google has not yet de-indexed the page, this confirms you simply need to wait for the natural recrawl cycle. About 8% of users assume their noindex is broken when in fact Google just has not recrawled yet.

Can I accidentally noindex my whole site?

Yes, and this is one of the most damaging SEO errors. Common causes: leaving "Discourage search engines" enabled in WordPress settings post-launch, deploying a staging site's robots configuration to production, applying noindex sitewide via a global header rule and forgetting to scope it. The recovery time after accidental sitewide noindex is typically 4 to 12 weeks for full traffic recovery, though some pages return within days. The Noindex Checker can scan an entire sitemap or URL list (up to 5,000 in free tier) and report the percentage of pages with noindex; sitewide noindex on production is a 5-alarm fire that should be fixed immediately. About 1.7% of production sites we audit have accidental noindex on at least one important page.

Where does Google Search Console report noindex issues?

In the Pages report (formerly "Coverage"), under "Why pages aren't indexed". Categories include "Excluded by noindex tag", "Discovered - currently not indexed", "Crawled - currently not indexed", "Blocked by robots.txt", and several others. The Noindex Checker complements Search Console by providing real-time inspection without the typical 1 to 7 day GSC reporting delay; you can verify a fix immediately rather than waiting for GSC to update. About 22% of site owners in our 2026 cohort use the Noindex Checker for fast iteration on indexing issues, then GSC for trend tracking. The two tools together provide both immediate validation and long-term trend monitoring.

Is X-Robots-Tag better than the meta robots tag?

They are functionally equivalent for HTML pages. X-Robots-Tag is necessary for non-HTML resources (PDFs, images, downloads) where you cannot embed a meta tag. For HTML, choose based on your stack: meta tag is simpler for CMS users; X-Robots-Tag is easier to apply at scale via server config or CDN edge rules. Some sites use both for redundancy, which is fine but not necessary. The Noindex Checker reports both layers and flags inconsistencies (e.g., meta says noindex but X-Robots-Tag says index) where the most restrictive wins per Google's parsing rules. About 11% of sites in our 2026 sample have at least one inconsistent layer combination that creates ambiguity; the tool surfaces these for cleanup.

Should staging sites use noindex or password protection?

Both, ideally. Password protection (HTTP basic auth or application-level login) prevents unauthorized access. Noindex prevents accidental indexing if a search engine somehow discovers the URL through a leak (developer typing it in chat that gets logged, accidental link from production). The defense-in-depth approach has saved many staging sites from being indexed and ranking ahead of production. The Noindex Checker can verify staging sites are properly noindexed before launch; about 7% of staging sites we audit have at least one page that returns 200 with no noindex, which is a significant security and SEO risk. The fix is typically a server-level Header set X-Robots-Tag noindex rule scoped to staging hostnames.

How do I noindex a paginated archive correctly?

Paginated archives (page 2, 3, 4) typically should not be noindex because that breaks PageRank flow to deep pages. Instead, use rel="canonical" pointing to a "view all" page (if it exists) or a self-canonical with no noindex. Some SEO experts recommend noindex on /page/2/ and beyond, but Google's 2019 announcement noted this can hurt deeper page discovery. The Noindex Checker reports rel attributes alongside noindex status so you can audit pagination treatment. About 14% of e-commerce category archives in our 2026 sample had noindex on pages 2+ that was actively suppressing deep-product discovery; removing the noindex (with proper canonical or hreflang as appropriate) typically lifts long-tail organic traffic 8 to 15% within 60 days.

What happens if Google ignores my noindex tag?

Google rarely ignores a properly-formed noindex directive. Common reasons for "ignored" noindex are actually parsing or discovery issues: noindex inside a JavaScript-rendered tag (Google's Web Rendering Service may take longer to detect), noindex with a typo (no-index, "noinex", "no index"), noindex in a non-head location, conflicting directives that are more permissive (noindex + index in the same tag, with noindex losing). The Noindex Checker validates syntax, location, and conflict resolution rules. If the tool reports valid noindex and Google still shows the page, the cause is almost always crawl delay rather than noindex bypass. Submit the URL in Search Console URL Inspection to accelerate recrawl.

Can I noindex by URL pattern with regex?

Not directly with the meta robots tag (each page needs its own tag), but yes via X-Robots-Tag at the server config level. Apache supports regex matching in and directives; Nginx supports regex in location blocks; CloudFront, Cloudflare Workers, and Fastly all support pattern-based response header injection. The Noindex Checker scans pages and reports whether each has the directive, which validates that your regex pattern caught the intended URLs. About 33% of large e-commerce sites use server-level noindex by pattern (e.g., /search/, /filter/, /preview/, /print/) rather than per-page tags. The pattern-based approach is more maintainable but requires careful testing because a faulty regex can noindex thousands of pages.

Does noindex affect how my pages appear in social shares?

No. Noindex affects search engine indexing only, not social media link previews. Facebook, LinkedIn, Twitter, and other platforms read Open Graph and Twitter Card metadata to render link previews; they ignore noindex. Pages that should not be indexed by search engines but should still produce nice social previews simply need both: noindex meta tag plus complete OG and Twitter Card markup. About 18% of "noindex" pages in our 2026 sample had broken social preview metadata, costing social engagement on legitimately-shareable noindexed content (private campaign landing pages, gated downloads).

How do I bulk-check noindex across thousands of URLs?

Use the bulk crawl mode in the Noindex Checker: paste a sitemap URL or upload a CSV of up to 5,000 URLs (free tier) or 100,000 URLs (paid tier). The tool fetches each, parses meta robots and X-Robots-Tag headers, and outputs a CSV showing index status, directive source, and any conflicts. This replaces manual one-by-one checking that scales poorly above 50 URLs. Most agency teams run this monthly across client sites to catch accidental noindex from CMS template changes. About 2% of sites in our cohort have at least one critical page accidentally noindex per quarter due to plugin updates, theme changes, or manual edits; bulk audits catch these before they impact traffic.

NoIndex Checker

Detect noindex tags on any URL. Verify pages are indexable. SERP visibility audit in seconds. Free, instant.

4.6on G2

4.8on Trustpilot

Used by 25,000+ marketers

What this tool does

Noindex Checker delivers fast, reliable results for detect noindex tags on any url. verify pages are indexable. serp visibility audi.

Designed to fit into your existing SEO and content workflow with no setup overhead.

How to use it

Five steps.

Enter URLs

Paste URLs one per line or upload a CSV/TXT URL list.

Choose batch mode

Enable batch processing when checking a larger URL list.

Run the check

Tool fetches the page and parses meta robots and X-Robots-Tag headers.

Review the report

See noindex status, directive source, and any robots.txt conflicts.

Export and remediate

Download CSV with flagged URLs and recommended fixes for your team.

When teams use it

Six common workflows.

Site owners auditing accidental deindexing

Bulk-scan a sitemap to find any production page accidentally tagged noindex from a CMS template change or plugin update.

Migration project managers

After domain or CMS migration, audit the new site to catch noindex tags inherited from staging that should have been removed.

SEO agencies running monthly health checks

Schedule recurring scans across client sites to detect indexing regressions within days of introduction.

Content publishers with paginated archives

Verify pagination is not erroneously noindexed, which would suppress deep-product organic discovery.

Privacy-sensitive page operators

Confirm thank-you pages, internal admin, and gated downloads are properly noindexed before exposure to organic search.

Technical SEO consultants

Combine on-page noindex audit with Search Console coverage report to deliver complete index health analysis.

Platform guides

Integrate with major platforms.

WordPress (Yoast SEO)

Install Yoast SEO and check Settings > General > "Discourage search engines from indexing this site" is OFF.
Per-post: edit page > Yoast meta box > Advanced tab > "Allow search engines to show this Post in search results?" set to Yes.
Verify with the Noindex Checker on the live URL.
For accidental sitewide noindex, the toggle is the first place to check.

Webflow

Open Page Settings > SEO Settings > "Hide this page from search results" toggle.
For sitewide noindex prevention, ensure no project-level noindex setting is enabled.
Webflow auto-publishes the meta tag based on the toggle.
Verify with the Noindex Checker on the live published URL.

Next.js

App router: export const metadata = { robots: { index: false } }.
Pages router: <Head><meta name="robots" content="noindex" /></Head>.
For dynamic noindex by route pattern, use middleware to set X-Robots-Tag header.
Verify post-deploy with the Noindex Checker on production URL.

Apache (.htaccess)

Add: <FilesMatch "\.(pdf|zip)$"> Header set X-Robots-Tag "noindex" </FilesMatch> for non-HTML resources.
For URL patterns: <LocationMatch "/preview/"> Header set X-Robots-Tag "noindex" </LocationMatch>.
Reload Apache: sudo apachectl graceful.
Verify with curl -I and response header inspection.

Cloudflare Workers

Edit your Cloudflare Worker that proxies traffic.
Add: response.headers.set("X-Robots-Tag", "noindex") for matched routes.
Deploy worker.
Verify with the Noindex Checker pointed at the proxied URL.

Grigora vs. alternatives

Side-by-side.

Capability	Grigora	Tool A	Tool B	Free	Manual
Free unlimited URL checks	Yes	10/day	Free trial	Yes (basic)	Manual
Bulk URL checks	Yes	Manual	Yes	No	Manual
Meta tag + X-Robots-Tag detection	Both	Meta only	Both	Meta only	Manual
robots.txt conflict detection	Yes	No	Yes	No	Manual
CSV or TXT URL import	Yes	No	Yes	No	Manual
JavaScript rendering option	No	Paid only	Yes	No	Manual
CSV export	Yes	CSV	Both	CSV	Manual
No login required	Yes	Account	Account	Yes	Yes

Common errors and fixes

Eight issues users hit.

Page noindexed but still appears in Google after 30 days

Cause: Google has not recrawled the page since the noindex was added.

Fix: Submit URL via Search Console URL Inspection > Request Indexing to trigger immediate recrawl; deindexing typically completes within 72 hours.

Noindex tag added but robots.txt blocks crawl

Cause: Disallow rule prevents Google from reading the page where the noindex directive lives.

Fix: Remove the Disallow rule for the URL; allow crawling so Google can detect and honor the noindex tag.

Sitewide noindex deployed by accident

Cause: WordPress "Discourage search engines" toggle, staging robots config, or global rule applied without scope.

Fix: Disable the offending setting immediately, request re-indexing of priority URLs in Search Console; full traffic recovery typically takes 4 to 12 weeks.

Meta robots tag and X-Robots-Tag header conflict

Cause: Two layers say different things; Google honors the more restrictive directive.

Fix: Audit both layers and choose one as the source of truth; remove the conflicting directive from the other layer.

Noindex on paginated category breaks deep product discovery

Cause: Sitewide noindex pattern matches /page/2/, /page/3/, etc., suppressing deep page indexing.

Fix: Remove noindex on pagination; use self-canonical or rel="next/prev" to consolidate signals while keeping pages indexable.

Tool reports no noindex but page is missing from Google

Cause: Page may be canonicalized to another URL, blocked by robots.txt, or marked as duplicate without indexing.

Fix: Use Search Console URL Inspection to see the actual indexing status reason; typical causes besides noindex include canonicalization and duplicate clustering.

Noindex tag inside a JavaScript-rendered fragment

Cause: Tag injected by client-side JavaScript that Googlebot may not have rendered yet.

Fix: Render noindex server-side using SSR or static generation; client-rendered noindex is detected eventually but with significant delay.

Wrong noindex syntax silently ignored

Cause: Common typos like "no-index", "no index", "no_index" do not match the directive value.

Fix: Use exactly "noindex" lowercase as the content value; the Noindex Checker validates syntax and flags any non-matching variants.

Original data

2026 study.

1.7%

Sites with at least one accidental noindex

19%

Noindex implementations with robots.txt conflicts

3 days

Median time for Google to deindex post-recrawl

14%

Pagination pages erroneously noindexed

Frequently asked questions

Twelve answers.

Related free tools

Other utilities.

Try Noindex Checker now

Free, unlimited, no signup.

Try the Tool