robots.txt did not keep Claude's shared chats out of Google — and it will not keep you out of AI answers either

The paragraph in Google's documentation that explains the whole incident

Google's own page on blocking indexing states it without hedging: “For the `noindex` rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the `noindex` rule, and the page can still appear in search results, for example if other pages link to it.”

robots.txt governs fetching, not listing. A URL Google has been told not to crawl can still be indexed on the strength of links pointing at it — and a share link pasted into Reddit, X or a public Slack is exactly such a link. Decrypt reported the resulting entries carrying Google's tell-tale “No information is available for this page” label, which is the signature of a URL indexed without ever having been fetched. The instruction that actually removes a page is `noindex`, delivered as a meta tag or an `X-Robots-Tag` header — and it only reaches the crawler on a page the crawler is permitted to fetch.

What claude.ai serves today

Two readings we took ourselves on 28 July 2026 at ~16:40 UTC, both HTTP 200:

• `https://claude.ai/robots.txt` (`content-type: text/plain`) still carries `Disallow: /share/` under `User-Agent: `, alongside `/chat/`, `/join/` and `/api/*` — plus a separate group handing `GPTBot`, `OAI-SearchBot` and `ChatGPT-User` a blanket `Disallow: /`.
• A `HEAD` on `https://claude.ai/share/` now returns `x-robots-tag: none` — `none` being the shorthand for `noindex, nofollow`, delivered as a header rather than a tag.

The header is the correct instrument. Note the tension it now sits in: by the paragraph quoted above, a URL still disallowed in robots.txt is a URL whose headers Google is not supposed to go and read. Removal requests and the ageing of already-cached entries are separate levers we cannot observe from outside, and Google's results were already clearing before either file could have mattered. We are reporting what the two files say today, not predicting where they land.

Now run the same mechanic in the direction you actually care about

Every business we audit eventually asks the inverse question: if I block the AI crawlers in robots.txt, am I out of AI answers? If I allow them, am I in? Our own corpus says no on both counts. The figures below come from our production audit database, cut 28 July 2026: 1,042 completed engine answers across 176 audits in four local verticals, of which 520 are ChatGPT answers. An answer counts once per domain no matter how many times it links it, so the unit is answers-citing-domain, not links. Both robots.txt files were re-fetched as plain text the same morning.

• Yelp gives `OAI-SearchBot` a `Disallow: /`, with narrow exceptions for `/spk/` and `/article`. ChatGPT cited yelp.com in 0 of 520 answers — the finding we took apart in our Yelp piece.
• MapQuest carries an explicit, dated “Block AI/LLM Crawlers and Training Bots — Last Updated: January 2026” section handing `Disallow: /` to `GPTBot`, `OAI-SearchBot`, `ClaudeBot` and `PerplexityBot`. ChatGPT cited mapquest.com in 12 of 520 answers, 20 links, across med spas, dentists and personal-injury law.

Same directive, opposite outcomes — which is the entire point. A link can reach an answer through a licensing arrangement, through the search index the engine sits on top of, or through a fetch the user triggered, and none of those is the training crawler your robots.txt is addressing. Blocking a crawler is a statement about crawling and training. It is not a statement about citation, and it has never been enforceable as one. That asymmetry is the same one Cloudflare quantified at 50,000-to-1 when it measured crawls against referrals.

Three things to do with this

1. 1To get something out of search: serve `noindex` (meta tag or `X-Robots-Tag`) and allow the crawler to fetch the page. Adding a robots.txt `Disallow` on top does the opposite of what you intend — Anthropic's weekend is the largest available demonstration.
2. 2To get into AI answers: treat robots.txt as table stakes, not a lever. Don't block the retrieval agents (`OAI-SearchBot`, `PerplexityBot`) if you want to be cited. Blocking the training crawlers (`GPTBot`, `ClaudeBot`, `CCBot`) is a separate and defensible choice, and MapQuest is the standing evidence that it does not automatically cost you citations.
3. 3Either way, stop inferring and measure the output. No configuration file tells you what an engine cites. Only the engine does, and the answer differs by engine and by question.

If you want to know which side of this you are on, run the free check — we ask ChatGPT and Perplexity the questions your customers actually ask and show you, verbatim, whether you were named and what got cited instead.