Robots.txt, Sitemap and Canonical: The Indexing Trio

How Google discovers and indexes pages

Before a page can show up in search, it goes through three distinct stages — and nearly every "why isn't my page on Google?" problem comes from blurring the line between them:

1. Discovery (crawling): Googlebot finds the URL, either by following a link from another page or by reading a sitemap.
2. Rendering: the bot downloads the HTML, runs the JavaScript, and builds the page the way a browser would.
3. Indexing: the content is analyzed and, if it clears the quality bar, enters the index — the giant database search results are pulled from.

Each stage has its own control. robots.txt acts on the first (discovery). The noindex tag acts on the third (indexing). The canonical tells Google which version of several near-identical pages should stand in for all of them. And the sitemap helps discovery by handing over the list of URLs on a plate. The three that headline this guide — robots.txt, sitemap and canonical — are exactly the levers you control on each of those fronts.

One concept worth keeping: crawl budget. Google doesn't have infinite time to visit your site. On small sites this rarely matters, but on a portal with tens of thousands of URLs, letting the bot waste time on junk pages (filters, sort orders, internal search) means fewer visits to the pages that count. The three files below are, at bottom, how you steer that attention.

robots.txt: what it does (and what it doesn't)

The robots.txt file is plain text that lives at the root of the domain — always at https://yoursite.com/robots.txt, never in a subfolder. It gives crawling instructions to bots: "you can come in here", "stay out of there". The syntax is lean:

Here's the part that trips up almost everyone: robots.txt blocks crawling, not indexing. They are different things. If you block /promo with Disallow but another site links to that URL, Google may still list it in results — just without a description, with that "No information is available for this page" note. The bot obeyed and stayed out, yet the URL exists and got indexed anyway.

Other gotchas: robots.txt paths are case-sensitive, it supports the wildcards * (any sequence) and $ (end of URL), and it's a public file — anyone can read yours. Never list a secret path there thinking you're hiding it; you're doing the opposite, pointing a sign straight at it. To build the file without botching the indentation or forgetting the sitemap line, the robots.txt Generator assembles the structure from the rules you pick.

XML sitemap: handing the crawler a map

If robots.txt says "where not to go", the XML sitemap says "here's everything that exists". It's a list, in XML, of the URLs you want discovered — especially handy for pages that are new, deep (many clicks from the home page), or poorly linked internally. The format is simple:

• <loc> — the full, canonical URL of the page (required).
• <lastmod> — the last-modified date in ISO 8601 (2026-06-30). Google uses this field when it's honest and consistent.
• <changefreq> and <priority> — frequency and priority. Google has stated it ignores both; you can fill them in, but expect no effect.

There are hard limits: each sitemap file holds at most 50,000 URLs or 50 MB (uncompressed), whichever comes first. Bigger than that? You split into several sitemaps and create a sitemap index — a sitemap that points to the others. Every URL listed should return a 200 status, be the canonical version, and not be blocked in robots.txt. A sitemap and robots.txt contradicting each other is a mixed signal that confuses the crawler.

For Google to find the file there are two routes: declare a Sitemap: line in robots.txt and/or submit it through Google Search Console (the old URL "ping" endpoint was retired in 2023). The Sitemap XML Generator builds the <urlset> with the right header and fields from your list of URLs.

Canonical: solving duplicate content

The same page is often reachable through several URLs without you realizing it. Look at how these four can serve identical content:

• https://shop.com/sneakers
• https://shop.com/sneakers/ (trailing slash)
• https://shop.com/sneakers?color=black&utm_source=email
• http://www.shop.com/sneakers (http and www)

To Google, that's four different URLs with essentially the same content. It dilutes signals: the links and authority that should reinforce one page get scattered across the variants. The fix is the <link rel="canonical" href="..."> tag in the <head>, pointing every variant at the "official" version. It's like saying: "this is the original — consolidate everything onto it".

Two things matter here. First, every page should carry a self-referencing canonical — pointing to its own clean form — even when there's no obvious duplication; it guards against tracking parameters and scraped copies. Second, the canonical is a hint, not an order: Google weighs the signal alongside others (internal links, the sitemap, redirects), and if those contradict each other it may pick a different canonical than the one you declared. Keep every signal consistent. The Canonical URL Generator outputs the tag already normalized, without the parameters that shouldn't be part of the official version.

hreflang for sites in more than one language

If your site has versions in, say, English and Portuguese — like this one, with /en/tools and /ferramentas — hreflang tells Google which version to show each audience, so the English page doesn't surface for someone searching in Portuguese (and vice versa). The annotation goes in the <head>:

• <link rel="alternate" hreflang="en" href="https://site.com/en/tools/x">
• <link rel="alternate" hreflang="pt-BR" href="https://site.com/ferramentas/x">
• <link rel="alternate" hreflang="x-default" href="https://site.com/en/tools/x">

Three rules that, if broken, make hreflang get silently ignored: the language code follows ISO 639-1 (en, pt) plus an optional region in ISO 3166-1 (en-US, pt-BR); the references must be reciprocal — if A points to B, B has to point back to A, and each page must list itself; and x-default marks the fallback version for languages you don't cover. Since the reciprocity error is the most common, generating the whole block at once helps — the hreflang Generator produces the complete, symmetric set for each pair of URLs.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

Not reliably. robots.txt stops crawling, but a blocked URL can still be indexed if other pages link to it — showing up in results with no title or description. To remove a page from the index, use the noindex meta tag and keep the page crawlable, so Googlebot can actually read the instruction.

Do I need a sitemap if my site is small?

It's not mandatory. Small sites with good internal linking get discovered without trouble. But a sitemap doesn't hurt and helps in three cases: brand-new pages, deep pages, and lightly linked content. Since it costs almost nothing to maintain, it's worth having.

Does a canonical guarantee Google picks the URL I chose?

No. The canonical is a strong signal, but it's a suggestion. Google cross-checks it against internal links, redirects and the sitemap. If everything points the same way, it usually honors your choice; if there's a contradiction, it may elect a different URL as canonical. Keep your signals consistent.

Can one sitemap list the versions in several languages?

Yes, and it's a valid alternative to hreflang in the HTML: the sitemap accepts xhtml:link annotations that declare each URL's alternate versions. The key is not to mix the two methods inconsistently — choose to put hreflang in the <head> or in the sitemap, and keep the reciprocity intact.

How often does Google re-read these files?

There's no fixed number. robots.txt is typically fetched about once every 24 hours (or more often on very active sites); sitemaps are reprocessed as Google crawls the site. Big changes — like unblocking a directory that was previously off-limits — can take anywhere from hours to a few days to take effect in the index.