An e-commerce site publishes 300 new product pages in one week. Two weeks later, half of them still don’t appear on Google. The XML sitemap was in place, the URLs were listed in it, and nothing was blocked by robots.txt. The problem lay between discovery and indexing—two stages that are often confused but follow distinct rules. Understanding what a sitemap actually does, and what it doesn’t, helps prevent this kind of situation.
Page in the sitemap but missing from the index: a three-step diagnosis
This is the most frustrating scenario. The URL appears in the sitemap, Google Search Console confirms that the file has been processed, and the page remains impossible to find with a site: query. Before changing anything, you need to pinpoint where the process is getting stuck.
Discovery, crawling, and indexing are three separate stages. The sitemap plays a role in the first: it signals that a URL exists. Crawling then depends on the crawl budget the search engine allocates to the site and the page’s technical accessibility. Indexing, finally, is an algorithmic decision based on the content’s quality and relevance.
To make the right diagnosis, carry out these three checks:
- Inspect the URL in Search Console or Bing Webmaster Tools. The report shows whether the page has been discovered, crawled, and indexed or excluded (with the specific reason).
- Check the page’s technical directives: a
noindextag, a redirect, a canonical pointing to another URL, or a block in robots.txt. Any of these can break the chain after discovery. - Check the internal linking structure. Even if it’s listed in the sitemap, an orphan page sends a signal that it’s of little importance. Search engines take internal links into account when deciding what to crawl first.
In Bing, the sitemap coverage report lets you distinguish between known URLs and those that have actually been indexed. The gap between the two columns gives you a direct measure of the problem.
For example, you can look at the Tech Planete sitemap to see how a file is structured in practice: URL hierarchy, update frequency, and whether the lastmod tag is present.

XML sitemap and crawling: what the file actually changes for Google
A sitemap doesn’t force indexing. It’s an inventory, not an instruction. Google reads it to discover URLs it might not find by following the site’s internal links. On a well-linked site with few pages, the file adds very little. On a large site with deep or recently created pages, it can measurably speed up discovery.
Each sitemap file can list up to 50,000 URLs, and a sitemap index can reference up to 50,000 files. This allows very large sites—such as news publishers, e-commerce stores, and directories—to declare their entire inventory.
The role of lastmod in crawl prioritization
The lastmod tag indicates when a page was last modified. Google says it uses this date to schedule recrawls of URLs it has already discovered, provided the date is reliable. If the date is automatically updated every time the sitemap is generated, without any actual change to the content, Google will eventually ignore it.
A reliable lastmod value speeds up recrawling of updated content. A misleading lastmod value undermines the search engine’s trust in the entire file. It’s best to update this date only when the page’s content has actually changed.
Google sitemap ping discontinued
On June 26, 2023, Google discontinued its ping notification mechanism. Calls to the old ping URL no longer have any effect. There are still two ways to submit a sitemap: through Search Console or by adding a Sitemap: line to the robots.txt file.
Bing, meanwhile, offers IndexNow to notify it when URLs are added, updated, or removed. This protocol complements the sitemap rather than replacing it: IndexNow reports individual changes, while the sitemap provides the full inventory. Microsoft notes that a successful IndexNow notification does not guarantee that a URL will be crawled or indexed.
Sitemap and internal linking: two complementary SEO mechanisms
It’s sometimes said that a sitemap “replaces” good internal linking. That’s a mistake in practice. A sitemap signals that a URL exists. An internal link passes authority and semantic context. The two work at different stages of the indexing process.
A page with no internal links but listed in the sitemap will be discovered, but rarely prioritized. Search engines interpret the absence of internal links as a signal that the page is of little importance. If no other page links to it, the crawl budget will go to other URLs considered more central.
In practice, use the sitemap to cover gaps in your internal linking:
- Recently published pages that haven’t yet received internal links from other content.
- Deep pages accessible only after several levels of navigation (pagination, filters, archives).
- Seasonal or temporary content that doesn’t warrant permanent inclusion in the main site structure.
The XML file passes neither popularity nor topical relevance. It says, “This page exists.” An internal link says, “This page matters in this context.” Both are necessary on a medium-sized or large site.

Check that your sitemap and technical directives are consistent
A common pitfall is listing URLs in the sitemap that the site otherwise prevents search engines from crawling or indexing. Including a page blocked by robots.txt, marked noindex, or redirected with a 301 to another URL sends the search engine conflicting signals.
The sitemap should contain only crawlable, indexable, canonical URLs. Before each submission, check that the listed URLs return a 200 status code, have no exclusion directive, and don’t point to a different canonical URL.
On a CMS such as WordPress, sitemap plugins sometimes include media file URLs, tag archive pages, or internal search results. These URLs use up crawl budget without adding value to the index. Regularly cleaning up the file and excluding these unwanted entries helps focus crawlers on the pages that matter.
A sitemap remains a simple but demanding technical tool: it works when every URL it contains is worth crawling and indexing, and when no technical directive contradicts that intent.



