How fast do AI crawlers fetch a new page compared with Googlebot? On our last seven posts OAI-SearchBot arrived inside six hours every time; Googlebot reached three of six in 11–34 hours and has not yet touched the other three

What we measured, and how

Seven posts went live after the ledger's birth: three on 08-17 (14:02Z, 18:27Z, 18:32Z), three on 08-18 (13:56Z and two in one deploy at 16:48Z) and one on 08-19 (13:57Z). For each we pulled every ledger row on that path and took, per crawler token, the first hit after the publish instant. The ledger stores only requests whose user agent matches a bot-shaped pattern (`bot`, `crawl`, `spider`, `fetch`, `-user`, and a few named agents), so a fetcher that presents a browser user agent is invisible to it; probe rows from our own verifier are excluded. Publish time is the `created` timestamp of the production deployment (`vercel inspect`), which precedes the page being servable by roughly 30–50 seconds of build. The three pings we send at publish are the ones most sites could send: a `POST` to `api.indexnow.org` (IndexNow's global endpoint, which the IndexNow FAQ says is shared with Bing, Yandex, Seznam, Naver, Yep and Amazon), a Search Console Sitemaps API resubmission of `/sitemap.xml`, and nothing to OpenAI, Perplexity or Anthropic — they have no ping. Six of the seven posts were at least 24 hours old at the read (24–51 h); the seventh was 2.7 hours old and is reported only for the minutes-scale row.

One validation worth stating: for the three posts Google has indexed, Search Console's `lastCrawlTime` matches the ledger's first Googlebot row to the second (2026-08-18T04:35Z, 05:46Z and 2026-08-19T04:26:47Z). The ledger and Google's own record agree on when Googlebot fetched the page.

First fetch per crawler, six posts ≥24 h old

The order was the same on every post: a search index that accepts IndexNow (Yandex, sometimes Bing) within minutes, then OAI-SearchBot inside six hours, then some subset of PerplexityBot, Amazonbot and Googlebot over the following day. Per post: the 08-17 14:02Z post went YandexBot +2 m → OAI-SearchBot +1.4 h → Googlebot +14.6 h → Amazonbot +16.1 h; the 18:27Z post YandexBot +1 m → OAI-SearchBot +5.3 h → Amazonbot +23.1 h → Googlebot +34.0 h → ChatGPT-User +39.3 h; the 18:32Z post bingbot +1 m → YandexBot +1 m → OAI-SearchBot +5.3 h → Googlebot +11.2 h → Amazonbot +27.1 h → PerplexityBot +29.7 h; the 08-18 13:56Z post YandexBot +3 m → OAI-SearchBot +22 m → PerplexityBot +10.3 h → Amazonbot +18.4 h; the two 16:48Z posts YandexBot +2 m / +25 m → OAI-SearchBot +5.6 h → PerplexityBot +7.4 h (→ Amazonbot +13.1 h on one). Googlebot has not fetched any of the three 08-18 posts at 24–27 hours; Search Console reads "Discovered – currently not indexed" on all three.

The sitemap is read in minutes; the page is not

Googlebot fetched `/sitemap.xml` within one to two minutes of every one of the six resubmissions (08-17 14:03Z, 18:29Z, 18:33Z; 08-18 13:58Z, 16:48Z; 08-19 13:57Z). So Google knew the URL existed almost immediately — and then scheduled the page fetch 11 to 34 hours later, or has not scheduled it yet. On a site with under a hundred /news URLs that is what "Discovered – currently not indexed" looks like from the server side: the discovery happened; the crawl is queued. IndexNow does not shorten that queue, because Google is not on IndexNow's endpoint list; it does get Yandex to the page in two minutes and, on two of seven posts, Bing in one. OpenAI's search crawler, which has no ping at all, reached every page faster than Google's. How it found them we can only partly see: OAI-SearchBot read `robots.txt` 17 times and the `/news` index once in the window and never fetched the sitemap, while GPTBot fetched the sitemap at 21:36Z on 08-17 and 21:40Z on 08-18 — in each case one to two hours before OAI-SearchBot fetched the posts published that afternoon, in pairs, seconds apart (23:48:05Z and 23:48:07Z; 22:24:36Z and 22:24:39Z). OpenAI's crawler documentation says that when both bots are allowed "we may use the results from just one crawl for both use cases to avoid duplicative crawling"; we report the timing as a correlation, not a mechanism. PerplexityBot's four fetches landed in a single sweep at 00:13Z on 08-19, seconds apart, with no sitemap fetch recorded — which is consistent with it learning the URLs from something other than our sitemap, and also consistent with a sitemap fetch under a user agent the ledger does not classify.

The one user-directed fetch

OpenAI's documentation distinguishes OAI-SearchBot ("used to surface websites in search results in ChatGPT's search features") from ChatGPT-User, which "is not used for crawling the web in an automatic fashion" — "when users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." The ledger holds 69 ChatGPT-User fetches across the 2.7-day window, 23 of them on the homepage (the fetch-and-cite pairing our self-audit reports beside ChatGPT's four homepage citations) and most of the rest on posts a week or more old. Exactly one of the 69 was on a post from this cohort: the ChatGPT retrieval post, at 09:43:56Z on 08-19, 39.3 hours after publish and 5.3 hours after Googlebot's first fetch, from a Netherlands egress. That is the whole of what "a new page reaching a ChatGPT user" looks like in this sample: one fetch, on one of seven pages, on day two. We do not know whether the answer cited it; the RESONEO study this post discusses found that opened pages were cited 74% of the time.

What this means for a business that cares about AI visibility

• For ChatGPT search, a new page is fetchable within hours, without a ping. Six of six posts were in OAI-SearchBot's hands inside six hours. If the page is meant to be the thing ChatGPT cites for a question, the fetch is not the bottleneck; what the page says and whether it gets selected is — which is why we measure citation, not crawl.
• Google's queue is the slow lane, and the sitemap ping does not skip it. Resubmitting the sitemap got the sitemap read in two minutes and the page read in 11–34 hours, or not yet. If your agency reports "submitted to Google" the day a page ships, that is the sitemap, not the page. The first click on a new post arrives after the index, not after the publish.
• The training crawlers did not come. GPTBot fetched the sitemap twice and no page; ClaudeBot fetched nothing. On this site, in this window, a robots.txt decision about GPTBot or ClaudeBot would have changed zero fetches of new content; the OAI-SearchBot and PerplexityBot decisions are the ones that touch what ChatGPT and Perplexity can cite tomorrow, and Amazonbot is the quiet fourth visitor on five of six posts.

Limits

Seven pages on one small site over 2.7 days. "Not fetched" is censored at 24–51 hours — Googlebot may arrive on the three 08-18 posts tonight, and the 08-17 pattern (Googlebot at 11–34 h) suggests it will. The ledger sees only bot-shaped user agents; a fetch presenting a browser string is not in these counts. Publish time is the deployment's creation time (±1 minute of servability). We resubmit the sitemap through the API on every publish, which most sites do not do, so the Googlebot-sitemap-in-two-minutes row is a property of that habit as much as of Google. The ChatGPT-User row counts fetches, not citations. We will re-read the same seven URLs next week and report Googlebot's completion and any further user-directed fetches; if you want the same read on your own site, a free check shows which engines currently cite you, and the full audit reports the crawler picture beside it.