XML Sitemaps: The Complete Guide to Getting Pages Indexed

A sitemap does not make Google index your pages. It makes Google aware of them. That distinction explains almost every frustrating conversation a product team has ever had about "we submitted the sitemap, why are only 40% of URLs indexed?"

This guide covers what a sitemap can and cannot do, how to structure one for a site with thousands of URLs, and how to read the discovery signals you get back.

What a sitemap is actually for

A search engine discovers URLs in two ways: by following links, and by reading a list you hand it. Link-based discovery is the primary channel, and it also carries a quality signal — a page linked from your global navigation is obviously more important than one buried four clicks deep. A sitemap carries no such signal. It is an inventory.

That makes sitemaps most valuable in exactly three situations:

• Weakly linked pages. New articles, seasonal landing pages, or product detail pages that only appear in a paginated list behind a filter.
• Large sites. Above roughly 10,000 URLs, crawlers will not reliably find everything by traversal alone within a useful timeframe.
• Sites with JavaScript navigation. If your menu renders through a framework router and emits no real href, link discovery may find almost nothing. (See JavaScript rendering and crawlability for the full picture.)

If your site is 200 pages with a clean HTML menu, the sitemap is a nice-to-have. If your site is a 40,000-SKU storefront, it is infrastructure.

Structure: one file, or an index

A single sitemap file is capped at 50,000 URLs and 50 MB uncompressed. Past that you need a sitemap index — a file that lists other sitemap files.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
    <lastmod>2026-08-28</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-09-01</lastmod>
  </sitemap>
</sitemapindex>

Split by content type, not arbitrarily. sitemap-pages.xml, sitemap-products.xml, sitemap-blog.xml is far more useful than sitemap-1.xml through sitemap-9.xml, because Search Console reports coverage per sitemap file. When product indexing drops 30%, a typed split tells you that immediately; a numbered split tells you nothing.

The four tags, honestly assessed

Tag	Does it matter?	What to do
<loc>	Essential	Absolute URL, matching the canonical exactly — same protocol, same host, same trailing-slash convention
<lastmod>	Yes, if honest	Set it to a real content-change timestamp. Search engines learn to distrust sites where every URL updates daily
<changefreq>	Effectively ignored	Safe to omit
<priority>	Effectively ignored	Safe to omit

The lastmod caveat is the one that bites teams. A CMS that stamps lastmod with the deploy time, or the current date, teaches the crawler that the field is noise — and then it stops being used as a recrawl hint on the pages where you actually need it.

Canonical consistency is the whole game

Every URL in a sitemap is a claim: this is a page I want indexed, at this address. Contradicting that claim elsewhere is the single most common reason pages sit in "Discovered – currently not indexed".

Audit these four things together, and make sure they agree:

1. The URL in the sitemap
2. The rel="canonical" on that page
3. The meta robots directive on that page
4. Whether robots.txt allows crawling that path

A sitemap URL that canonicalises to a different address is a wasted entry. A sitemap URL blocked by robots.txt is worse — the crawler cannot read the page to discover the noindex you also set, so the URL can linger in the index without content. If you want a page out, allow the crawl and serve noindex; never block it in robots.txt and assume that is the same thing.

What belongs in the sitemap — and what doesn't

Exclude anything that is not a canonical, indexable destination:

• Redirecting URLs (3xx) and error pages (4xx/5xx)
• Pages canonicalised to another URL
• noindex pages
• Paginated pages beyond page 1, in most cases
• Faceted filter combinations — the classic crawl-budget sink (crawl budget optimization goes into this)
• Parameterised duplicates: ?sort=, ?utm_, session IDs

Include only the addresses you would be happy to see as a search result.

Multilingual sites: hreflang in the sitemap

For a site with Korean and English versions, you can declare language alternates inside the sitemap rather than in every page's <head>:

<url>
  <loc>https://example.com/en/pricing</loc>
  <xhtml:link rel="alternate" hreflang="en" href="https://example.com/en/pricing"/>
  <xhtml:link rel="alternate" hreflang="ko" href="https://example.com/ko/pricing"/>
  <xhtml:link rel="alternate" hreflang="x-default" href="https://example.com/en/pricing"/>
</url>

The rule that catches everyone: annotations must be reciprocal and self-referencing. Each URL lists every alternate including itself. A one-directional declaration is discarded. The hreflang guide covers the failure modes in detail.

Submission and monitoring

Reference the sitemap from robots.txt so any crawler finds it:

Sitemap: https://example.com/sitemap.xml

Then submit it in Google Search Console and Naver Search Advisor. After that, the number worth watching is not "submitted" — it is the ratio of submitted to indexed, per sitemap file, over time.

A healthy pattern: indexed count climbs toward submitted count within a few weeks and then tracks it. An unhealthy pattern: submitted grows, indexed flatlines. That gap is almost always quality or duplication, not discovery — and no amount of resubmitting fixes it.

A practical audit routine

Once a month, or before any release that touches URLs:

1. Crawl the site and export the full URL inventory. (Our DeepAnalyzer produces this alongside an IA sheet.)
2. Diff the crawl against the sitemap. URLs in the crawl but not the sitemap are discovery gaps; URLs in the sitemap but not the crawl are usually orphans or stale entries.
3. Check the status code and canonical of every sitemap URL. Anything not 200-and-self-canonical comes out.
4. Compare the submitted/indexed ratio per sitemap file against last month.

That loop catches the vast majority of indexing regressions before a stakeholder notices traffic moving.

FAQ

Does submitting a sitemap guarantee indexing?

No. A sitemap is a discovery aid. Indexing depends on whether the page is crawlable, canonical, and considered worth storing. Pages that are thin, duplicated, or heavily templated frequently stay unindexed no matter how often the sitemap is submitted.

How often should a sitemap be regenerated?

Whenever the URL inventory changes. For a CMS or storefront, generate it dynamically so publishing a page updates the sitemap in the same deploy. A static file regenerated quarterly is worse than no file, because it teaches crawlers to ignore your lastmod.

Should paginated pages be in the sitemap?

Usually only page 1. The remaining pages should be reachable by crawling and should carry self-referencing canonicals, but listing them all inflates the sitemap without improving discovery of the underlying items.

What size should I split a sitemap at?

The hard limits are 50,000 URLs and 50 MB uncompressed, but split much earlier — by content type — so coverage reporting is diagnostic. Ten typed files of 5,000 URLs are far more useful than one file of 50,000.

Is an HTML sitemap page still worth building?

For users, sometimes. For crawlers, a well-structured global navigation plus an XML sitemap does the job better. Build the HTML version only if real visitors use it.