Where do AI crawlers actually go on a small site? In 7 days ChatGPT-User fetched our homepage 193 times (71% of its requests), Googlebot spent 56% of its requests on robots.txt and sitemap.xml, and Amazonbot read 151 distinct pages — more than any AI fetcher

Method, stated before the numbers

Every request to askedabout.com whose User-Agent looks like a bot (matches `bot|crawl|spider|slurp|fetch|-user|…`) is forwarded by the site's edge middleware to a Convex table, with the path, the UA, and the edge's country code. A fixed token table then names the fetcher by substring — `ChatGPT-User`, `OAI-SearchBot`, `GPTBot`, `PerplexityBot`, `Perplexity-User`, `ClaudeBot`, `Claude-User`, `Amazonbot`, `Googlebot`, `Google-Extended`, `bingbot` and about a dozen more. A bot-shaped UA that matches no token is stored as "other" with its UA intact, never dropped. We described the ledger and its fetch-speed read in an earlier post.

This cut pulls the trailing seven days (2026-08-19 13:53 to 2026-08-26 13:37 UTC, 6.99 days) — 2,035 rows. Two kinds of row are excluded and counted: 58 are our own healthcheck's labelled probe requests, and 22 are requests for secret-shaped paths (`/.env`, `/.git/HEAD`, `/etc/passwd`…) made under AI-crawler User-Agents, which we showed on 2026-08-22 are a vulnerability scanner wearing the badge. That leaves 1,955 net requests across 172 distinct paths: 1,227 from fetchers the token table names and 728 from bots it does not.

Each path is then assigned one class by a fixed rule, in this order: the homepage (`/`); plumbing (`/robots.txt`, `/sitemap.xml`, `/llms.txt`, `/favicon.ico`, `.well-known`, `humans.txt`, `ai.txt`); a `/news/` post; the `/news` index; the `/compare`, `/guide`, `/data`, `/for` and `/library` collections; a site page (`/about`, `/terms`, `/privacy-policy`, `/refund-policy`, `/pricing`, `/agency-5-pack`…); anything else is unclassified (11 requests, all listed in the artifact). "Content pages" below means everything except the homepage, plumbing and unclassified. A hit is one HTTP request; nothing is de-duplicated by IP, and — this matters — a User-Agent that claims to be Googlebot is counted as Googlebot unless it asked for a secret-shaped path. The classifier, the class rule and every count are in the linked script and artifact.

The allocation, fetcher by fetcher

Three things in that table are the finding. First, ChatGPT-User — the fetcher OpenAI uses when a live ChatGPT answer opens a page on a user's behalf — is almost entirely a homepage reader here: 193 of 272 requests (71%), 0 plumbing, and 59 requests spread over 23 of our 108 fetched posts; the most-fetched post drew 10. On this site the homepage is also where 9 of 9 ChatGPT citations of us have landed across four weekly gauge runs, so the fetch pattern and the citation pattern agree. Second, Googlebot's week was mostly bookkeeping: 99 of 176 requests were robots.txt (80) or sitemap.xml (19), and the remaining 71 content fetches touched 45 of 151 live URLs — 24 of them /news posts, in a week when we published 8 and had 20 URLs sitting unindexed. Third, the widest reader is not an answer engine: Amazonbot fetched 151 distinct content pages including 105 of the 108 posts, one request each, 97% content and 0% plumbing, against PerplexityBot's 67, ClaudeBot's 66 and OAI-SearchBot's 57.

Two smaller readings. ClaudeBot's 139 requests came on 4 of the 8 calendar days, 97 of them on Monday 2026-08-24, and exactly half were the sitemap (32) and robots.txt (32) — it re-reads the map, then reads 44 posts once each. GPTBot, OpenAI's training crawler, made 8 requests: 7 for robots.txt and 1 for the homepage, 0 content pages — on this site the OpenAI reader that matters is OAI-SearchBot (129) and the OpenAI reader that matters most is ChatGPT-User.

The 728 requests from bots the classifier does not name

37% of the week's bot requests came from fetchers our table does not name, and they are a different population: SEO-tool crawlers (AhrefsBot, SemrushBot, MJ12bot, DotBot), Huawei's PetalBot (203 requests across 148 paths, the single busiest reader of the week on any list), IbouBot, and a tail of small "AI citation" crawlers (GEO-Brain-CitationBot, CitableBot, SpyglassesBot, ShapBot) that fetch a handful of pages each. These read content pages 70% of the time. None of them is an engine that can cite a page to a buyer; all of them are in the denominator of any "AI crawler traffic" number that counts bot-shaped User-Agents without naming them.

Where the week's requests went, all fetchers

Named fetchers put 24% of their requests into plumbing and 19% into the homepage before any content page is read; the /news class, which holds 107 of the site's 151 live URLs, drew 37%. The four commercial classes together (`/compare`, `/guide`, `/data`, `/for`, 37 live URLs) drew 16% — 196 named-fetcher requests, of which PerplexityBot alone made 59.

Limits

• One small site, one week, 1,955 requests. The allocation is a description of this ledger, not of the crawlers' behaviour in general; a site with a different shape (more product pages, fewer posts) would read differently.
• A request is counted by the User-Agent it presents. We exclude the scanner rows we can identify by path, but a spoofed Googlebot or GPTBot asking for an ordinary page is counted as the real thing; we do not verify IP ranges.
• ChatGPT-User's homepage fetches say that a live ChatGPT answer opened our homepage; they do not say whether the answer then cited it, mentioned it, or discarded it. Our citation gauge is a separate instrument.
• The ledger was born 2026-08-17, so this is its first full-week cut; the 8 "calendar days" are UTC dates with partial ends (08-19 from 13:53, 08-26 to 13:37).
• 108 distinct `/news/` paths were requested against 107 live posts — at least one request was for a path that no longer resolves; the class rule counts it as a post request.

What we will read next Wednesday

The same cut over the next seven days (2026-08-26 13:37 to 09-02, run on our 09-02 growth run) is pre-committed on three numbers. (A) ChatGPT-User's homepage share stays at or above 50% of its requests, else "ChatGPT reads the homepage" is falsified for this site; below 30% we say so at the top. (B) Googlebot's plumbing share stays at or above 40%, else the "mostly bookkeeping" reading is falsified. (C) Amazonbot's distinct content paths remain at or above each of PerplexityBot's, ClaudeBot's and OAI-SearchBot's, else "the widest reader is not an answer engine" is falsified as stated. Each is a number in next week's artifact, not an interpretation.

The script is `content/news/crawler_allocation_cut.mjs`; the artifact is `content/news/crawler-allocation-cut-2026-08-26.json` (9 of 9 self-checks: the probe/spoofed/net partition, every fetcher's classes summing to its requests, class and per-day totals summing to the net, UA families summing to the unnamed bucket), and the slimmed row dump (timestamp, vendor, token, kind, path, country, UA family — no full UA strings) is `content/news/crawler-rows-slim-2026-08-19-to-26.json`. If you run a small site and want the same read, the ledger is 120 lines of middleware and one table. If you would rather know what the engines say about your business than which of their crawlers visited, the check on our homepage asks them.