Which Pages Should Not Be in Your Sitemap: 12 Types

Start with Google’s sitemap guidance: list the versions you want to appear in search. The twelve types below are a practical audit list, followed by the exceptions and the action each type needs.

Which pages should not be in your sitemap: the 12 types

Find your URL type and read across. The third column is what to use instead.

A temporary 5xx error is a repair task, not a reason to retire a valuable URL. Check the page and server first. Keep wanted pages eligible for the sitemap once they recover. Likewise, an empty or unavailable product page needs an availability decision before it is treated as permanently gone.

The four-question test that decides every URL

You do not have to memorise twelve types. A sitemap is a list of nominations, not an inventory, so the question is whether you would nominate this URL. Four gates, in order.

1. Does it return a successful page? Normally 200, with the real content rather than an error message. Fix temporary failures; list the final URL after a permanent move.
2. Is this your preferred canonical version? Keep the sitemap, internal links, redirects and canonical annotations consistent.
3. Can the crawler access and index it? A wanted page with an accidental noindex, crawl block or login requirement needs that problem fixed. An intentional exclusion stays out of the sitemap.
4. Is it a useful search landing page? If not, leave it out. Use noindex to keep an accessible page out of search; use crawl rules to control unwanted URL spaces. Those actions have different purposes.

This test is for XML sitemaps. An HTML sitemap for visitors is an ordinary web page with its own indexing decision. Google ignores priority and changefreq; it uses lastmod when that date reliably reflects a significant page update. Do not stamp every URL with the build date. These details are covered in the sitemap documentation.

Why prune a sitemap: clearer signals and useful reports

For most small sites, the immediate benefit is a reliable list of wanted URLs and an indexing report you can act on. Crawl-budget management becomes more important on very large or rapidly changing sites, or when many URLs remain discovered but not indexed.

Google’s crawl-budget guide gives rough examples of a million-plus URLs changing weekly or 10,000-plus changing daily, rather than hard thresholds. It also covers sites with many discovered, unindexed URLs. A small site whose new pages are crawled promptly should focus on useful content, current sitemaps and indexing diagnostics.

The page on myths about crawling goes further. Pages that serve a 4xx status, with 429 the exception, "don't waste crawl budget" at all. And "Improving your crawl rate won't necessarily lead to better positions in Google Search results."

Two reasons survive, and neither is about budget.

• Every URL you list is a nomination. Google's canonicalisation guide says "All pages listed in a sitemap are suggested as canonicals", a "weak signal" but a signal. List a duplicate and you argue against the page you want ranking.
• Search Console fills with errors you caused. The Page indexing report documentation names it as a cause of error spikes: "you might have submitted a sitemap that includes URLs that are blocked for crawling by robots.txt, noindex, or a login requirement."

A messy sitemap is not itself proof of a penalty. It can create inconsistent canonical preferences and clutter the report with URLs you never intended to index. Fix those specific problems rather than expecting a ranking increase from deleting lines.

What to do with each type instead of listing it

Duplicates and non-canonical URLs

This is the group that costs you something. A www and a non-www copy, a trailing-slash twin, one product under three category paths. Pick one URL per piece of content, point the rest at it with a 301 or rel=canonical, and list only the one you picked. The canonical checker shows what a page actually claims, which is often not what its owner assumes.

Pages you have already blocked or noindexed

Keep intentionally excluded URLs out of the sitemap. The rule may be a meta robots tag in the page HTML or an X-Robots-Tag response header. A page you do want indexed needs an accidental directive fixed instead. Noindex and an unlisted URL are not privacy controls. Protect a private blog or NDA portfolio with authentication; anyone who obtains a public URL can still open it.

Broken and moved URLs

404s, soft 404s and anything answering a 3xx. Google's advice is narrower than most people assume: "In general, we recommend fixing only 404 errors that you link to yourself or list in a sitemap." A sitemap entry is you vouching for a URL, so a dead one is yours to fix. The redirect checker gives you the URL at the end of the chain. List that one.

Private and transactional pages

Cart, checkout, order confirmation, thank-you, account, password reset. None work as a search entry point, and a thank-you page in the index collects people who skipped everything meant to convince them. Policy pages are the exception people get wrong: your terms and privacy pages are ordinary indexable pages and belong in the file.

Machine-made URLs: search, filters and calendars

Filters can create a large URL space: four choices in each of five filters already allow 1,024 combinations. For variants you do not need indexed, Google’s faceted-navigation guidance recommends controlling crawling. For an existing indexed URL you want removed, noindex must first be accessible to the crawler. Preserve a selected facet that serves a distinct search need, has useful content and a consistent canonical URL; a query parameter alone does not disqualify a page.

Archives you never chose to publish

WordPress core includes public post types, taxonomies and author archives in its sitemap system. Audit the pages themselves. An empty tag archive or an author archive that merely repeats the blog index may add little; a curated topic page or substantial author profile may be useful. Neither the folder name nor the number of posts alone decides. Keep useful, indexable archives and exclude the ones you intentionally noindex.

Staging, previews and other hosts

The sitemaps protocol says "all URLs in a Sitemap must be from a single host", so staging and preview URLs have no place in your live file. If a staging copy is turning up in search, the fix is a password on that host, not an edit to the file.

noindex, robots.txt, 410 or canonical: which one you need

Choose the action by the job it must do. Sitemap exclusion changes the submitted URL list; noindex controls search inclusion; robots.txt controls crawling; authentication controls access; redirects and canonicals help consolidate duplicate or moved content.

The trap is combining the first two. Google's guide to blocking indexing with noindex spells it out: "If the page is blocked by a robots.txt file ... the crawler will never see the noindex rule, and the page can still appear in search results." The Page indexing report even has a status for the result: Indexed, though blocked by robots.txt. Leave the page crawlable until the rule is read, and expect to wait: Google warns it "may take months for Googlebot to revisit a page".

For a large unwanted crawl space, a robots.txt rule can reduce crawling. It is not a dependable way to remove a URL already in search. If an accessible page must remain available to visitors but leave search, use crawlable noindex. If it is permanently gone with no relevant replacement, return 404 or 410. If it moved, use a relevant permanent redirect. The robots.txt generator and meta robots tag generator help with syntax after you choose the right action.

Five pages people wrongly leave out

Pruning has its own failure mode. These five get cut by people working down a checklist, and every one is a page a sitemap is useful for.

• Useful category, tag, author and collection pages. A curated topic hub or substantive author profile can be a good search landing page. Judge its content and purpose instead of excluding a whole archive type by name.
• Orphan and weakly linked pages. Some guides say exclude orphans, which is backwards: discovery is one of the few jobs a sitemap genuinely does. Run the hub page that should point there through the internal link finder, add the missing link, and keep the URL listed.
• Page 2 onwards of a paginated sequence. Do not automatically noindex them or canonicalize them all to page 1. Google recommends a distinct URL and self-canonical for each page. Eligible pages can be listed; sitemap inclusion is not a requirement to list every pagination URL.
• PDFs and downloads you do want found. A price list, a spec sheet, a sample chapter. Media gets cut as a category when the question is per file.
• Brand-new pages with nothing linking to them yet. This is the moment a sitemap earns its keep, and part of why a blog helps your SEO: every new post starts with no inbound links.

Audit your sitemap in 15 minutes

Five free tools, then the report that tells you whether any of it worked.

1. Confirm the file loads. Paste your domain into the sitemap status checker. It finds the sitemap and reports its address and whether it is available, which is worth knowing before you start deleting lines from it.

View the saved sitemap availability check at full size.

2. Read what is in it. The sitemap visualizer lists every URL in the file with its last-modified date and tells you how many there are. Read down it for /tag/, /author/, ?sort= and anything on a staging host.

3. Bulk-check directives and fetch errors. The noindex checker accepts URLs or a CSV and reports meta robots, X-Robots-Tag and robots.txt findings. Treat an error separately from an intentional noindex. Its “indexable” result describes detected technical eligibility, not confirmation that Google or Bing has indexed the page. Verify important URLs in the relevant webmaster tool.

View the saved Noindex Checker report at full size.

4. Test the rule before you ship it. The robots.txt tester takes your live file or an unpublished draft and quotes the exact line behind each verdict.

5. Regenerate, if you write the file yourself. The XML sitemap creator crawls your site to a depth of 1 to 5 levels, up to 10,000 URLs, with include and exclude path filters for the patterns you found.

Then finish in Search Console, where the Page indexing report filters to a single sitemap. Google's tip there is worth stealing: to speed up a fix validation it suggests you "create and submit a sitemap containing only your most important pages, then filter the report by that sitemap". A twenty-URL sitemap of the pages that pay the bills tells you more than one with five thousand lines.

How to exclude a page on WordPress, Yoast, Shopify and Grigora

On most platforms the sitemap is generated. Change the source settings, regenerate or republish, then open the resulting XML to verify the change. Editing a generated file by hand may last only until the next rebuild.

WordPress core supports filters for post types and taxonomies. A developer can exclude a provider or disable core sitemaps, but removing a sitemap entry does not by itself add noindex to the page. Check which plugin or core feature actually generates your file before changing it.

With Yoast SEO installed, the plugin takes over the file and builds it from what you let search engines index. Its documentation gives the example that "Post Types marked as noindex will not appear in the sitemap." So you exclude something by setting it to noindex in Yoast, not by keeping a second list in sync.

In Rank Math, Sitemap Settings → General has separate fields to exclude post/page IDs and taxonomy term IDs; its Authors settings control author archives. Rank Math’s instructions show the settings. Decide separately whether the underlying URL should remain indexable: a sitemap exclusion setting and noindex are different actions.

On Shopify every store generates a sitemap.xml with "links to all your products, primary product image, pages, collections, and blog posts", and it updates itself as you add things. You exclude items at the source. Shopify's guide to hiding pages from search engines says an Unlisted product is hidden from the sitemap, and a seo.hidden metafield set to 1 hides a page, blog post or product from sitemaps and search engines. Crawl rules for filter and search URLs go in the robots.txt.liquid template.

On Grigora, publishing generates the sitemap from eligible pages. Settings → SEO → Sitemap Configuration includes Add Author Pages in Sitemap and Add Category Pages in Sitemap. Set these according to whether those pages are useful search entry points; a single author is not, by itself, a reason to hide the author profile. The Blog Sitemap Frequency and Blog Sitemap Priority fields write optional values Google ignores. After publishing, inspect the generated XML and confirm that intentionally noindexed pages are absent. Try Grigora free for 14 days, no card needed, or start from a website template.

What removing a URL from your sitemap does not do

Removing a sitemap entry does not remove a URL from Google or stop crawling: the engine may already know it from previous crawls or links. Google’s Sitemaps report documentation makes the same distinction when a sitemap is removed from the report. Apply the indexing or access control the page actually needs.

The Removals tool is not the answer either. Google says requests there last for about 6 months, and that permanent removal means removing or updating the content, password-protecting the page, or adding noindex. Not robots.txt, for the reason above.

If a URL must leave search promptly, choose the permanent action first, then use temporary removal when appropriate:

1. Sensitive or private content: remove it or require authentication immediately. Noindex leaves it publicly accessible.
2. A public page that should remain available: add noindex and allow the crawler to read it. A permanently deleted page can return 404 or 410 immediately; there is no need to wait for noindex first.
3. Remove intentionally excluded URLs from the sitemap and update links as appropriate. Redirect only to a relevant replacement when a page has moved.
4. For urgent Google suppression, use Search Console’s temporary Removals tool alongside the permanent action. Its temporary block lasts about six months; verify the URL status afterward rather than assuming that submitting the request completes removal.

The cheaper version is deciding what should be visible before anything goes public. Our list of things to do before others can view your blog covers that, and it is shorter work than unpicking it later.

Sitemap exclusion FAQs