Once AI Memory has a sitemap, it turns the URLs into a page list, sorts every page into a page type and crawls the pages worth reading. This article explains each step and the limits that apply, so you can predict what a crawl will do on a client's site.
From sitemap to page list
Every page URL from the sitemap is stored in the page list on the Sitemap tab.
AI assigns each page a page type.
The enabled pages are crawled. When the crawl was started from Accounts, this starts on its own after the pages are sorted. On the Sitemap tab you can start it with Crawl enabled, which shows the number of pages waiting.
On the Sitemap tab you can filter by page type (including Uncategorized), language and crawl state, turn pages on or off, and change a page's type. After a page is crawled, its type is refreshed from its actual content, unless someone set the type by hand.
Page types
There are 30 page types: Homepage, Product, Service, Article / Blog, Listing / Overview, Market / Audience, Landing, FAQ, Contact, Event, Job Posting, About, Pricing, Recipe, Profile, How-To Guide, Legal, Location, Local SEO, Comparison, References, Brand, Partners, Cornerstone, Thank You, Login, Basic Page, Offer, Gallery and Other.
The page type decides what is extracted from a page. Product and Service pages also fill the Products and Services cards in the company profile.
Pages that are not crawled by default
These types are detected and listed, but turned off for crawling when a page is first sorted, because they add noise rather than knowledge:
Thank You
Login
Profile
Local SEO (city or doorway landing pages, which are near-duplicates by design)
You can turn any of these pages on again. A page you turn back on stays on.
Pages that never become memories
These types are crawled and stay searchable for the AI, with the page itself as the source, but no memories are created from them:
Legal
Thank You
Login
Local SEO
Article / Blog
Blog posts and news are time-sensitive. A figure in an old announcement would otherwise be treated as a current fact, so company facts come from the lasting pages instead.
How a page is fetched
Cache. A copy of the page crawled in the last 30 days is reused.
Crawler service. Otherwise the page is fetched through a crawler service that can render pages which need a browser.
Direct fetch. If the crawler returns an empty page or a consent wall, fails, or doesn't finish within about 60 seconds, the page is fetched directly with a regular browser user agent.
Clean-up. Menus, headers, footers and other boilerplate are removed before the content is stored.
Because pages are not all fetched with the same user agent, allowing one bot name in a firewall is not enough. Make sure public pages load without a challenge page.
Speed and load on the client's server
By default, 6 pages are crawled at a time.
No more than 4 requests go to the same host at once, or 2 for Google and Blogger-hosted sites.
When a host answers with HTTP 429, every request to that host pauses. Requests are retried with growing waits (up to 30 seconds per wait), and a
Retry-Afterheader is respected. A page that stays rate-limited is kept in the queue instead of being marked as failed.
Large catalogs
When one page type has more than 500 pages, only 5 pages per category or subcategory are crawled. The rest are marked On demand and fetched when they are needed. The Sitemap tab shows how many pages are on demand, and they don't count as unfinished work, so the crawl can complete.
Re-crawling
On the Sitemap tab:
Re-crawl > Re-crawl & re-extract content fetches and extracts every enabled page again, using the cache.
Re-crawl > Clear cache & re-crawl skips the cache and fetches fresh copies.
Re-fetch sitemap reloads the URL list from the sitemap.
The sitemap also refreshes on the schedule set on the Sitemap tab. When a refresh finds new pages, the earlier approval is cleared so the new pages get reviewed, and the company's memory status returns to To verify.
The blog rule
Crawling a blog pulls in posts, not the company's real pages. The platform warns you when the address is:
a subdomain starting with
blog,blogs,news,nieuws,actualites,actusorartikels. For example, fornieuws.example.beit suggestsexample.be.a hosted blog platform: blogspot.com, wordpress.com, medium.com, tumblr.com or substack.com. This check runs on the Sitemap tab.
In Accounts, the Crawl button asks before crawling a blog subdomain. On the Sitemap tab, the This looks like a blog dialog offers the main domain instead.