Canonical, noindex, robots.txt and Sitemaps: Indexing Signals

What robots.txt, noindex, canonical and XML sitemaps do and don’t do: conflicts often seen in the field, a scenario-based decision table and validation with Search Console.
Technical summary
Technology
robots.txt (RFC 9309), meta robots, X-Robots-Tag, rel=canonical, XML sitemap, Google Search Console
Topic
Choosing indexing control signals correctly and using them together
Level
Intermediate
Prerequisites
  • Access to the site’s URL structure and templates
  • Access to the server or CDN configuration (for X-Robots-Tag)
  • A verified Google Search Console property
In this article7 sections

“We blocked the page with robots.txt but it still shows up on Google.” Many indexing complaints start with this sentence. Whether a page appears in search results isn’t determined by a single setting but by four signals that do different jobs: robots.txt, noindex, canonical and the XML sitemap. The problem comes from using these signals in place of each other or giving conflicting messages for the same URL. This guide explains what each signal actually does, which one to choose in which scenario and how to verify the result in Search Console.

How indexing control tools differ

The basic distinction is between crawling and indexing. Crawling is the bot requesting the URL and downloading the content; indexing is the content being added to the search index. Each signal touches a different link in this chain. Mix up the link and the result gets mixed up too.

Signal Stage it affects How binding it is Typical use
robots.txt Disallow Scanning Binding for compliant bots Infinite URL spaces, internal search, admin paths
meta robots / X-Robots-Tag noindex Indexing Binding (if crawled) Thank-you pages, thin archives
rel="canonical" Choosing between copies Hint Parameterized, tracking-coded, duplicate URLs
XML sitemap Discovery and priority Hint List of canonical URLs you want indexed
301/308 redirect URL consolidation Strong signal Pages moved permanently or merged

The robots.txt protocol has been standardized as RFC 9309; but it binds only bots that comply with the protocol and isn’t an access control mechanism. Content that must stay private is protected with authentication. robots.txt is public and announces the paths you block to the world. It isn’t a security tool.

Don’t confuse robots.txt with noindex

The mistake we see most often in the field is blocking a page with robots.txt to remove it from results. A blocked page can’t be crawled, so the noindex tag on it can’t be read either. If Google finds links to the page, it can index the URL without content; Search Console shows this in the Pages report as “Indexed, though blocked by robots.txt”.

The right order is: first noindex is added to the page, the page is left crawlable, and once it’s verified that it has dropped out of the index, a crawl block is added if needed. Google hasn’t supported the Noindex: rule in robots.txt since 2019. That line is dead code today.

For non-HTML resources (PDF, DOCX, images), noindex is given with an HTTP header. An example in Nginx:

location ~* \.(pdf|docx?)$ {
    add_header X-Robots-Tag "noindex" always;
}

When you add add_header to a location block in Nginx, the add_header directives in parent blocks (e.g. security headers) aren’t inherited in that block; repeat them if needed. This behavior is the classic reason security headers silently disappear from PDF responses. The equivalent in Apache with mod_headers enabled:

<FilesMatch "\.(pdf|docx?)$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

When canonical works, and when it’s ignored

Canonical means “this content has more than one address; this is the address I prefer”. It isn’t an order but a hint. Google weighs the declaration together with other signals and makes its own choice if it sees inconsistency. Typical situations where canonical is ignored:

  • Different content: A category page’s canonical has been set to the home page. If the pages aren’t really copies, the canonical loses its meaning.
  • Conflicting signals: The canonical points to A, but all internal links and the sitemap contain B.
  • Problem target: The canonical target returns 3xx, 404 or noindex.
  • Multiple canonicals: The theme and the SEO plugin each output a canonical; if there are several different values, Google may ignore them all.
  • Outside the head: The canonical is in the body, or the head has closed early because of an invalid element.

Canonical and noindex aren’t used together on the same page. When the messages “this page is a copy of B” and “don’t index this page” sit side by side, the consolidation signals can be lost because of the noindex. If it’s a copy, canonical. If it’s unwanted, noindex.

Canonical is written with an absolute URL and must be consistent with the form your corporate website prefers in terms of protocol, domain and trailing slash:

<link rel="canonical" href="https://www.example.com/services/technical-seo/" />

Parameterized URLs, filters and pagination

Parameters come in three types, and each is handled differently:

  1. Those that don’t change content (utm_source, gclid, session ID): The canonical points to the URL without parameters. These parameters are never used in internal links.
  2. Those that narrow content (filters, sorting): Valuable combinations (e.g. “category + brand”) can be indexed with their own canonical URLs; for the others, canonical or noindex is used. If the number of combinations explodes (multiple filters, price ranges), narrowing the crawl space with robots.txt preserves crawl resources.
  3. Pagination (/page/2/, ?page=2): Each page gets a self-referencing canonical. Google no longer uses rel="prev"/rel="next" markup as an indexing signal; normal <a href> links between pages are enough.

The old URL Parameters tool in Search Console has been removed; parameter management is now entirely down to site-side signals. The most reliable way to see which parameters are actually crawled is the server logs. Don’t guess, check the logs. The method is in the log-based crawl analysis article.

An example of narrowing the filter space with robots.txt:

User-agent: *
Disallow: /*?*orderby=
Disallow: /*?*filter_
Disallow: /?s=
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

The * wildcard and the $ end-of-line marker are defined in RFC 9309. When several rules match a URL, the longest (most specific) matching rule applies; on a tie, Allow wins. Don’t block CSS and JavaScript files; if Google can’t render the page, it evaluates the content as incomplete. Disallow: /wp-includes/ lines left over from old templates still cause this problem.

Content and maintenance of the XML sitemap

A sitemap is the list of URLs you want indexed; it isn’t “everything on the site”. Content rules:

  • Only URLs that return 200, don’t carry noindex and have a self-referencing canonical.
  • No redirected, deleted, parameterized or robots.txt-blocked URLs.
  • lastmod is updated only when content changes meaningfully. Google uses lastmod when it’s consistently accurate; it ignores the priority and changefreq values.
  • At most 50,000 URLs and 50 MB uncompressed in a single file; larger sites use a sitemap index file.

Since WordPress 5.5, core generates a sitemap at /wp-sitemap.xml. When a plugin such as Yoast SEO or Rank Math enables its own sitemap (/sitemap_index.xml), it usually disables the core sitemap. Still, check that the two sitemaps aren’t live at the same time and that only the one you use has been submitted to Search Console.

To audit sitemap cleanliness regularly, check URL status codes in bulk:

curl -s https://www.example.com/post-sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' \
  | while read -r u; do
      printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"
    done | grep -v '^200 '

The output should be empty. If lines come back, the sitemap has non-200 URLs, and those are cleaned up first.

Scenario-based decision table and validation

Scenario Recommended signal To avoid
Thank-you page after form submission noindex, outside the sitemap robots.txt block
Campaign URLs with UTMs Canonical to the URL without parameters Internal links to the parameterized URL
Internal site search results robots.txt Disallow (if not in the index) Adding to the sitemap
Permanently moved page 301/308 redirect Canonical only
Author or tag archive, thin content noindex, follow links preserved Canonical to the home page
PDF catalog with a counterpart web page X-Robots-Tag noindex or HTTP canonical Trying robots meta on the page
Test environment Authentication + noindex robots.txt only

Validation with Search Console

  1. URL Inspection: For a URL, compare the User-declared canonical and Google-selected canonical fields. If they differ, the signals are conflicting.
  2. Pages report: Examine these rows, whose names vary by interface language: “Excluded by ‘noindex’ tag”, “Indexed, though blocked by robots.txt”, “Duplicate, Google chose different canonical than user” and “Alternate page with proper canonical tag”. The first and last rows may be expected states; the middle ones usually need fixing.
  3. Sitemaps report: Examine the reasons for the difference between the number of submitted URLs and the number indexed in the Pages report, with the sitemap filter.
  4. robots.txt report: Check the latest version Google fetched and any fetch errors with the robots.txt report under Settings.

These signals break most often during site migrations and redesigns; the checks for the transition period are in the guide on preventing SEO loss when migrating a website. Structured data, on the other hand, isn’t an indexing signal but a separate layer that conveys content meaning; that topic is in the article on schema.org markup on a corporate site. Designing these signals together with the site architecture is the first thing we tackle in technical SEO and site architecture work.

Frequently asked questions

Why does a page blocked with robots.txt still appear in search results?

robots.txt blocks crawling, not indexing. If Google finds external or internal links to the page, it can index the URL without reading its content. If you want the page removed from results, the robots.txt block is lifted and noindex is added to the page.

Why did Google choose a different URL even though I wrote a canonical tag?

Canonical is a hint. If internal links, the sitemap, redirects and content similarity support a different URL, Google can make its own choice. Compare the user-declared and Google-selected canonical fields in the URL Inspection tool, and consolidate the signals on the same URL.

Should I keep a noindex page in the sitemap?

No. The sitemap contains only the canonical URLs returning 200 that you want indexed. Putting noindex pages in the sitemap creates conflicting signals and leads to unnecessary warnings in Search Console. The only exception is keeping it temporarily so the noindex is noticed quickly.

In paginated lists, should the second and later pages have the first page as their canonical?

Usually not. Since each paginated page lists different content, it gets a self-referencing canonical. Giving a canonical to the first page can weaken the discovery of links to items on later pages.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Web

We build and maintain corporate websites, WordPress and CMS platforms, intranet and SharePoint portals, document management, and form and approval workflows, and keep them secure and up to date.

Knowledge Center

Other articles