Naming an AI crawler in robots.txt is not the same as blocking it — we parsed 70 rules and 34 of them were not full blocks

What was wrong with the previous measurement

Our 2026-07-28 piece, robots.txt is not an AI-visibility switch, used a guarded fetcher that detects whether a robots.txt names an AI crawler. That is a real signal, and it was enough to make that post's point. It is also, on its own, misleading in a specific way: a `User-agent: GPTBot` header tells you a site operator thought about GPTBot. It does not tell you what they decided. `Disallow: /` and `Allow: /` sit under identical headers.

So this pass parses the rules. For each AI agent we now report one of four states: BLOCKED_ALL (`Disallow: /` with no `Allow` narrowing it), PARTIAL (a root disallow with path exceptions, or path-scoped disallows only), ALLOWED (a group exists and imposes no effective disallow), and NO_GROUP. Nine agents are checked per domain: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, anthropic-ai, PerplexityBot, Google-Extended and CCBot. The code is `ops/robots_rules.py` in our repo and the guard is unchanged from 07-28: a robots.txt counts as measured only on HTTP 200 and `content-type: text/plain` and a body containing no `<html`. Anything else is UNMEASURED and is never reported as a block — that guard exists because an unguarded probe once read Reddit's HTML network-block page as `Disallow: /`.

The result: 70 rules, and most of them are not what the header implies

Only 51% of the rules a group-presence check would have flagged are actual full blocks. Three cases show why that matters:

• yelp.com — the case we flagged in advance. GPTBot, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended and CCBot get `Disallow: /` with nothing else. But OAI-SearchBot gets `Disallow: /` plus `Allow: /spk/`, `Allow: /article$` and `Allow: /article/`, and ChatGPT-User and PerplexityBot get the `/article` carve-outs. Yelp did not block OpenAI's search crawler; it fenced it into a corner of the site.
• medicalspalocator.com — named all eight agents in our 07-28 pass, which reads like a wall. The actual rule for every one of them is `Allow: /` with `/admin/`, `/api/`, `/thank-you`, `/unsubscribe/` and `/spa-dashboard/` disallowed. That is an ordinary housekeeping robots.txt, not an AI policy at all.
• discovermedspa.com — also listed 07-28 as naming the agents. Its rule for all seven is a bare `Allow: /`. The site went out of its way to name the AI crawlers in order to let them in. Reading the group header alone, we had this exactly backwards.

Now pair the rules against who actually gets cited

The corpus is a fresh production export taken this morning: 1,048 completed answers across 177 audits and four verticals (dentists, med spas, home services, personal-injury law), 27 synthetic smoke rows dropped, 1,112 distinct registrable domains. As we noted on 07-29, this instrument reads in-text URLs and that is effectively a ChatGPT cut — 523 of the answers are ChatGPT's, and the citation counts below are answers-citing-domain out of those.

mapquest.com blocks all nine agents outright and is still cited in 12 answers. yelp.com blocks six and is cited in none. Same directive class, opposite outcomes. Meanwhile the four most-cited domains in the entire corpus — birdeye.com, bbb.org, threebestrated.com, reddit.com — do not name a single AI crawler, in either direction. If robots.txt were the switch, that column would sort. It does not.

The honest reading, and we will not go further than the data does: a link can reach an answer through a licensing deal, through the search provider's index, or through a user-triggered fetch, and robots.txt speaks to only one of those paths. We are reporting a pairing, not a cause. What we can say flatly is the negative: on this corpus, a full `Disallow: /` for every named AI agent does not prevent a domain from being cited, and naming no agent at all does not prevent it either.

What to do with this

• Read the rule, not the header. If you or a vendor are auditing a client's AI exposure by grepping robots.txt for `GPTBot`, you will report blocks that are allows. In our sample that error would have hit 34 of 70 rules, including a domain that names the crawlers specifically to welcome them.
• Treat a 403 on robots.txt as unknown, not as a block. Two of twenty domains refused us. A tool that quietly codes those as “blocked” is inventing a policy on behalf of a site that never stated one.
• Do not sell robots.txt edits as an AI-visibility fix. Unblocking a crawler is cheap and occasionally right, but nothing here supports the claim that it produces citations. The only way to know whether a change moved anything is to ask the engines the buyer's question before and after, and count.

That before-and-after is what our product does: the free 60-second check asks ChatGPT and Perplexity the way a customer would and reports how often a business is named ($79 adds Gemini and Claude, 25 questions sampled three times each). If you audit AI visibility for clients, the agency 5-pack runs it across five of them — and how to check whether your business shows up in AI walks the method if you would rather do it by hand.