Three of three crawler readings we pre-registered on 2026-08-26 held in week 3: ChatGPT-User 71% homepage (bar 50%), Googlebot 41% robots and sitemap (bar 40%), Amazonbot 166 distinct content pages against the AI fetchers' best of 69

What was pre-registered, and what the ledger says

The week-1 post (2026-08-26) closed with three readings to be scored on 2026-09-02 from a fresh 7-day dump, before the data existed. (A) ChatGPT-User sends at least half of its requests to the homepage; falsified below 30%. (B) Googlebot spends at least 40% of its requests on robots.txt and sitemap.xml. (C) Amazonbot touches at least as many distinct content pages as any single AI fetcher. The dump is 2026-08-26 13:37 to 2026-09-02 13:30 UTC: 2363 rows, 85 verifier probes excluded, 0 spoofed, 2278 net requests over 176 distinct paths. It does not overlap week 1; it overlaps week 2 (which ran 08-23 to 08-30) by four days, stated here because the series is a rolling read.

3 of 3 held. Reading (B) is the one to watch: 41% clears a 40% bar by 0.5 points, down from 56% in week 1 and 46% in week 2. Googlebot made more requests each week (176, 240, 269) and put more of them on content (71, 121, 152 content requests over 45, 55, 56 distinct pages). On this trend the reading fails in week 4 or 5, which is the point of writing the bar down first.

Method, unchanged from week 1

The site's edge middleware forwards every bot-shaped User-Agent to a Convex table with the path, the token that matched, and a timestamp; no IP is stored in the artifact. A fixed token table names the fetcher by substring, so identity is claimed by the User-Agent string, not proven. A request is one HTTP request; nothing is de-duplicated. Path classes are assigned by a fixed regex in the script: homepage is `/`; plumbing is robots.txt, sitemap files, llms files and the well-known set; content is every `/news`, `/compare`, `/guide`, `/data`, `/for`, `/library` and site page. "Distinct content pages" counts unique paths in the content classes. The script is `crawler_allocation_cut.mjs`, the same file for all three weeks; the week-3 artifact passes 9 of 9 self-checks (partition sums, per-fetcher class sums, the 7-day span, and unit cases for the path and User-Agent classifiers).

Three weeks, eleven fetchers

Windows: week 1 2026-08-19 13:53 to 2026-08-26 13:37 UTC (1955 net), week 2 2026-08-23 14:06 to 2026-08-30 13:39 UTC (2146 net), week 3 2026-08-26 13:37 to 2026-09-02 13:30 UTC (2278 net). Named fetchers were 1227, 1532 and 1612 of those; the rest are bot-shaped User-Agents matching no token (PetalBot, SemrushBot and AhrefsBot lead that bucket every week).

Three movements. ClaudeBot went from 139 requests at 46% plumbing to 169 at 91%: it now polls robots.txt and the sitemap and reads 15 pages a week, down from 66. YandexBot went from 55 to 202 requests, 68% of them plumbing. ChatGPT-User grew from 272 to 309 requests with the homepage share flat at 71%, 74%, 71%; it touched 34, 36, 40 distinct content pages. GPTBot is unchanged at 8 requests a week, 7 of them robots.txt, 0 content pages in any week.

Pre-registered re-read, 2026-09-09

Same script, the ledger window 2026-09-02 13:30 UTC to 2026-09-09, scored in the growth run that day. Three readings, written before the data exists.

1. 1(A) unchanged: ChatGPT-User homepage share at least 50%; falsified below 30%.
2. 2(B) tightened to the trend: Googlebot plumbing share stays at or above 40%. The stated expectation is that it does NOT: the series is 56%, 46%, 41%, and a fourth reading below 40% ends the "Googlebot spends most of its budget on plumbing" line in this series. A reading at or above 40% keeps it.
3. 3(C) unchanged: Amazonbot distinct content pages at least equal to every single AI fetcher's. The nearest AI fetcher this week is at 69; a fetcher crossing 166 would fail it.

Limits

• The ledger sees requests that reach the edge middleware. Cached responses never reach it, so every count is a floor.
• Fetchers are named by User-Agent, not reverse DNS. The known spoof-path scanner shape is excluded and counted; this week it was 0 rows.
• Week 2 and week 3 overlap by four days; week 1 and week 3 do not overlap. The three-week table is a rolling read, not three independent samples.
• The bars were set on week 1 alone, and (B) was set 16 points under the week-1 value. A bar that holds because it was set loosely is a weaker result than one set at the margin; the 09-09 read states the expected failure for (B) so the next result carries information either way.
• Artifacts: `content/news/crawler-allocation-cut-2026-09-02.json` (9 of 9 self-checks) from `crawler-rows-slim-2026-08-26-to-09-02.json`; the week-1 and week-2 artifacts are `crawler-allocation-cut-2026-08-26.json` and `crawler-allocation-cut-2026-08-30.json`. This post's text and tables were generated from the three artifacts by `crawler_allocation_week3_post.mjs`.