AI search news ·
Does Googlebot still fetch new posts on publish? 8 posts, zero Googlebot visits at 24–75 hours — OAI-SearchBot hit all 8 inside 12
Every post this site published between August 18 and August 20 — eight pages — has zero Googlebot fetches on our server-side crawler ledger, at ages 24 to 75 hours. The same ledger shows OAI-SearchBot fetching all eight within 12 hours of publish, PerplexityBot within 18, and YandexBot or bingbot reaching seven of the eight within 3 minutes. Googlebot itself is not away: it made 122 requests to this site over the ledger's 4.6 days — but every content page it touched was published August 17 or earlier. Here is the per-post ledger, Google's own documentation for the states Search Console reports, and the one control that shows the pipe still works when Googlebot does fetch.
The ledger, post by post
| Post | Published (UTC) | Age at read | Googlebot | OAI-SearchBot after | Search Console state at read |
|---|---|---|---|---|---|
| 82%-single-URL citation study | 08-18 13:56 | 74.6 h | 0 fetches | 22 min | Discovered – currently not indexed |
| Search Console logging-bug post | 08-18 16:48 | 71.7 h | 0 fetches | 5.6 h | Discovered – currently not indexed |
| llms.txt crawler-census post | 08-18 16:48 | 71.7 h | 0 fetches | 5.6 h | URL is unknown to Google |
| 79% source-reuse study | 08-19 13:56 | 50.6 h | 0 fetches | 9.9 h | URL is unknown to Google |
| Crawler fetch-speed post | 08-19 16:49 | 47.7 h | 0 fetches | 7.0 h | Discovered – currently not indexed |
| Reddit citations post | 08-19 16:49 | 47.7 h | 0 fetches | 7.0 h | URL is unknown to Google |
| Buried-pages referral post | 08-20 13:55 | 26.6 h | 0 fetches | 11.2 h | Discovered – currently not indexed |
| IbouBot/PetalBot post | 08-20 16:45 | 23.7 h | 0 fetches | 8.4 h | Discovered – currently not indexed |
Method: the ledger is a server-side log of user-agent-classified crawler requests (probe rows excluded), born 2026-08-17T01:48Z; publish instants are production-deployment creation times (±1 minute); Search Console states are from the URL Inspection API, read around 16:30 UTC today. PerplexityBot fetched all 8 at 7.2 to 17.9 hours, Amazonbot 7 of 8, and the first conventional search-engine bot (YandexBot or bingbot) arrived at 1 to 25 minutes — 7 of 8 inside 3 minutes. This extends the fetch-speed measurement we published August 19, where Googlebot fetched the 08-17 cohort at 11 to 34 hours: since August 18 the lag is no longer a lag, it is an absence. The machine-readable cut is saved alongside this post.
Googlebot is here — it just is not fetching anything new
In the same 4.6-day window Googlebot made 122 requests to this site: 51 for robots.txt, 11 for sitemap.xml (most recent at 16:24 UTC today), and 60 page fetches across 33 distinct paths — at least 9 of those page fetches happened today. Every one of the 33 paths was published 2026-08-17 or earlier. So this is not a crawler that stopped visiting; it is a crawler that keeps re-reading the back catalog, and the sitemap that lists the new URLs, without fetching the new URLs themselves.
The control: when Googlebot does fetch, indexing still follows
One post broke the pattern this morning — in the right direction. Our August 13 post on AI traffic in analytics had been stuck for 8 days across 8 consecutive non-indexed Search Console reads, twice regressing to 'URL is unknown to Google'. Today at 09:38:02 UTC the ledger recorded its first Googlebot fetch; the URL Inspection API reports lastCrawlTime 09:38:03 — the same second — and by this afternoon's read the state was 'Submitted and indexed'. The ledger and Search Console agree to the second when a fetch actually happens, which is what makes the zeros on the other eight rows a real absence of crawling rather than instrument blindness.
What Google's own documentation says
Google's Page Indexing report documentation defines 'Discovered – currently not indexed' as: "The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl." The URL Inspection documentation defines 'URL is unknown to Google' as "Google hasn't seen this URL before." Both sit oddly beside the ledger: a site receiving 26 Googlebot requests a day is hard to overload with one more fetch, and three of the eight URLs Google now reports never having seen are listed in a sitemap Googlebot fetched 11 times in five days — two of them read 'Discovered' yesterday. The states are also churning: four of the eight flipped between 'Discovered' and 'unknown' between yesterday's inspection and today's, in both directions.
Why this matters if you care about AI visibility
- The freshness order has inverted for this site. Retrieval crawlers for ChatGPT and Perplexity now read new pages in minutes to hours; Google has not fetched anything published in the last 4 days. One of the unindexed posts got a ChatGPT-User fetch at 15:08 UTC today — an assistant opened a page for a real user that Googlebot has never requested. New content can be working in AI answers while Google still reports the URL as unknown.
- Do not diagnose indexation from Search Console states alone. Our states swapped between 'Discovered' and 'unknown' on four of eight URLs in one day while the server-side fact — zero Googlebot fetches — stayed constant. A server log (or CDN log) is the ground truth; the inspection state is a weather report.
- IndexNow does not reach Google, and sitemap fetches are not crawls. Bing and Yandex — the engines that accept IndexNow — reached 7 of 8 posts inside 3 minutes. Google only takes the sitemap, fetches it promptly, and, on this ledger, has spent 4 days not visiting the URLs inside it.
Limits
One small, 70-day-old site — n=8, and 'Discovered – currently not indexed' is a common state for new sites with little authority; Google promises no crawl schedule, and nothing here shows a policy change rather than ordinary crawl-budget triage of a low-priority site. The ledger is 4.6 days old and classifies by self-declared user agent without IP verification. The observation is censored: any of the eight could be fetched an hour after this publishes — we will report the first Googlebot fetch and indexation of this cohort, with timestamps, as we did for the control above. If AI engines are now the first readers of your newest pages, the question that matters is what they say when asked about your business — a free check reads that directly.
See your number
See which businesses AI names when your client's buyers ask.
Running this for clients? The $249 agency 5-pack audits five businesses, white-labeled.
Who runs this
- Built and operated by Sensara LLC, Atlanta, Georgia — about us and how the audit works.
- See what the report looks like before you run anything — score per engine, the competitors AI names instead of you, and a fix plan.
- We run the same audit on ourselves every week and publish the result: in the latest run AI named AskedAbout in 1 of 144 answers. We report our own numbers the way we report yours.