Use sitemaps to support a sound directory crawl system
A technical workflow for deciding eligible URLs, producing accurate sitemap entries, preserving status and canonical integrity, and investigating coverage.
The practical answer first
A directory sitemap should contain only canonical, indexable, successful URLs with honest last-modified values. It supports discovery but does not replace crawlable internal links or guarantee indexing. Keep drafts, private routes, search combinations, redirects, and errors out; use sitemap indexes when scale exceeds protocol limits.
For developers, SEO teams, and operators responsible for large or frequently changing listing collections and multiple publishing domains.
Use this playbook to define sitemap eligibility, validate raw XML, and investigate crawl or indexing symptoms without making claims from impressions alone.
Define one URL eligibility function
A URL belongs in the sitemap only when it is published, canonical, allowed for indexing, and expected to return a final 200 response. Apply the same eligibility to directory roots, meaningful categories, and listing details. Do not include client app, login, submission workflow, private, draft, or unknown routes.
- Normalize hostname, case, slash, and encoding.
- Exclude redirects and alternate canonical variants.
- Remove retired records in the next generated response.
Generate standards-compliant XML
Escape XML values, use absolute canonical locations, and provide valid W3C timestamps only when content changed materially. Respect 50,000 URL and uncompressed size limits by using sitemap indexes and stable partitions when necessary. Keep customer-domain sitemap URLs on the customer canonical host.
- Return application/xml with a successful status.
- Use fixed release dates for unchanged static routes.
- Avoid setting lastmod to request time on every fetch.
Connect robots, links, and status behavior
Reference the correct sitemap from robots.txt, but do not use robots rules as an index-removal shortcut. Important pages need crawlable anchors from hubs, categories, or contextual content. Unknown or removed URLs must not receive a generic 200 application shell, and private routes need durable noindex protection.
- Test raw 404 responses for unknown slugs.
- Keep sitemap, canonical, and visible breadcrumb hosts aligned.
- Do not block resources required to render public content.
Validate and investigate evidence
Fetch and parse sitemap output, compare it with the source registry or database, sample every template, and check duplicate locations. Use Search Console sitemap and inspection data when available, but absence of impressions does not prove a URL is unindexed. Record exact property, date, URL, and verdict for any claim.
- Automate exact static-route parity tests.
- Sample newest, oldest, sparse, and removed listings.
- Monitor generation failures and unexpected count changes.
Sitemap release workflow
Run this workflow whenever route registries, canonical hosts, directory publication states, or sitemap partitioning change.
- 01Select
Query or enumerate canonical published URLs through one eligibility rule; exclude private, draft, redirect, and error routes.
- 02Render
Escape XML; use absolute locations; emit honest valid lastmod; partition below protocol limits; set XML headers.
- 03Compare
Assert exact static-route parity, database counts within limits, no duplicates, no private URLs, and stable hosts.
- 04Sample
Request representative locations for final status, canonical, robots, initial content, schema, and internal links.
- 05Observe
Monitor sitemap errors and count changes; use property-scoped search evidence; record uncertainty and release dates.
Frequently asked questions
Does submitting a sitemap guarantee indexing?
No. A sitemap helps discovery and communicates canonical candidates, but search engines still evaluate crawl access, canonical signals, content quality, duplication, and broader indexing policies.
Should every filter combination be in the sitemap?
No. Include stable pages that serve distinct intent and provide useful content. Unbounded combinations can create duplicates, empty pages, and crawl waste.
How should lastmod be set?
Use the time of a meaningful content or canonical-page change, not the current request time. Static guide releases can use a fixed honest date until the content materially changes.