The sitemap tells AI Memory which pages a website has. Everything after it (page types, the crawl, the extracted memory) starts from that list, so a missing or partial sitemap is the most common reason a memory looks thin. This article explains how the sitemap is found and read.
When the sitemap is looked up
Automatically, when someone clicks Yes, crawl my website on the company setup page (the button shows "Detecting sitemap…") or Crawl in the Memory column of the Companies list in Accounts.
By hand, when someone enters a sitemap URL on the Sitemap tab in AI Memory.
If a sitemap is already registered for that domain, clicking Crawl again does not register a new one. It continues crawling the pages that are not crawled yet.
Discovery order
AI Memory tries every host variant of the website address: first the address as entered, then its www or non-www twin. For an http address it also tries the https versions. For each variant it checks, in this order:
Blogger. If the site answers like a Blogger site, it uses
/sitemap-pages.xml, the only Blogger sitemap that lists static pages.robots.txt. Every
Sitemap:line. Whole-site translation sitemaps such asfr_fr-sitemap.xmlare tried after the site's own sitemap, unless their language matches the memory's language./sitemap_index.xml/sitemap.xml/sitemaps.xml/wp-sitemap.xml/page-sitemap.xml/sitemap-pages.xml/pages-sitemap.xml
A candidate only counts when it returns sitemap XML. A sitemap index wins over a flat sitemap found earlier in the list. If no sitemap is found at all, the website address itself is registered and the lookup is tried again when the sitemap is fetched.
Which sitemap gets registered
A sitemap index with one page sitemap and nothing else that might hold content: that page sitemap is registered directly, for example
.../page-sitemap.xml.Any other sitemap index (several page sitemaps, or product or custom sitemaps next to them): the index itself is registered and read as described below.
No index anywhere: a flat sitemap whose file name looks like a pages sitemap is preferred. Otherwise the first flat sitemap found is used.
How a sitemap index is read
AI triage. For each child sitemap, AI looks at the file name and up to 4 sample URLs from inside it. Each child becomes one of three things: pages (read and indexed), categories (kept as the shop's own category structure, not crawled as content) or skipped (blog posts, news, authors, tags, product attributes and filters, date archives, attachments and media).
Safety rules. Well-known page and product sitemaps (such as
page-sitemap.xml,sitemap-pages.xml,wp-sitemap-posts-page-1.xmlandproduct-sitemap.xml) are always kept. Attachment and media sitemaps are always skipped. If the AI step is unavailable, these file-name rules decide on their own.Category sitemaps, such as WooCommerce product category sitemaps, contribute up to 300 URLs each.
Nesting. Indexes inside indexes are followed up to three levels deep.
Size. Up to 50,000 URLs are read from one registered sitemap. Large catalogs are sampled at crawl time instead. See How the app crawls and categorizes pages.
A flat sitemap that you enter by hand is used as it is: every page URL in it is kept.
What is read from each entry
The page URL (
<loc>) and, when present, the last-modified date (<lastmod>).The page language, taken from the URL path, from the page's own
hreflangalternate link, or from a language in the sitemap file name such assitemap-nl-be-pages.xml.CDATA sections and XML entities are decoded.
Links to non-HTML files are skipped: images, PDFs, archives, audio and video, CSS, JavaScript, JSON, XML, text files and fonts.
Sitemap requests use the user agent UpToDateConnect-Bot/1.0, with a 5-second timeout for robots.txt and 10 to 15 seconds for sitemap files.
Setting the sitemap yourself
Open AI Memory for the company and go to the Sitemap tab.
Enter the exact sitemap URL, for example
https://yourwebsite.com/page-sitemap.xml. Add more than one URL to build a single page list.Choose how often it refreshes: Never, Daily, Every 3 days, Weekly or Monthly.
Click Save & Fetch.
Use Re-fetch sitemap to reload the list later. If approval is refused because no products or services were found, Look for more sitemaps checks the site's sitemap index again.
If the address looks like a blog subdomain or a hosted blog platform, the This looks like a blog dialog suggests crawling the main site instead, with a button to use the main domain, a button to crawl the blog anyway, and Cancel.
Make sure a client's sitemap is found
List the sitemap in robots.txt with its full https URL.
Use a sitemap index that keeps pages, products and posts in separate child sitemaps. Yoast, Rank Math and WordPress's built-in sitemaps do this.
Make sure the sitemap loads for automated visitors without a challenge page or consent wall.
Register the live domain, not a staging address.