robots.txt did not keep Claude's shared chats out of Google — and it will not keep you out of AI answers either

What changed after this post: the /share/ disallow was gone by 30 July

On 28 July we wrote that the two files on claude.ai worked against each other. The robots.txt told crawlers not to fetch /share/*, and the x-robots-tag: none header that takes a page out of search only works on a page a crawler may fetch. On 1 October 2026, between 16:39 and 16:42 UTC, we read both again and pulled every copy of the robots.txt the Internet Archive has captured since June. The disallow is gone. It is in the archive copies of 23 June and 10 July and in our own fetch of 28 July, and it is absent from the archive copy of 30 July at 20:23 UTC. Anthropic removed it inside those two days.

The header stayed. On 1 October a HEAD request to https://claude.ai/share/ returned HTTP 200 with x-robots-tag: none, and the answer was the same for a browser user agent and a Googlebot user agent. https://claude.ai/public/artifacts/ returned the same header. A GET on /share/ also carried <meta name="robots" content="noindex, nofollow"> in the HTML. We did not request any real shared conversation. With the disallow removed, Googlebot may fetch a share URL and read the noindex rule on it, which is the setup the Google paragraph quoted below describes.

What this reading does not show. We did not query Google, so we cannot say what its index holds for claude.ai/share today. We found no Anthropic statement about the robots.txt change. The archive holds five captures since June, so the removal is dated to a two-day window and no closer.

The paragraph in Google's documentation that explains the whole incident

Google's own page on blocking indexing states it without hedging: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”

robots.txt governs fetching, not listing. A URL Google has been told not to crawl can still be indexed on the strength of links pointing at it — and a share link pasted into Reddit, X or a public Slack is exactly such a link. Decrypt reported the resulting entries carrying Google's tell-tale “No information is available for this page” label, which is the signature of a URL indexed without ever having been fetched. The instruction that actually removes a page is noindex, delivered as a meta tag or an X-Robots-Tag header — and it only reaches the crawler on a page the crawler is permitted to fetch.

What claude.ai served on 28 July 2026

Two readings we took ourselves on 28 July 2026 at ~16:40 UTC, both HTTP 200:

• https://claude.ai/robots.txt (content-type: text/plain) still carries Disallow: /share/* under User-Agent: *, alongside /chat/*, /join/* and /api/* — plus a separate group handing GPTBot, OAI-SearchBot and ChatGPT-User a blanket Disallow: /.
• A HEAD on https://claude.ai/share/ now returns x-robots-tag: none — none being the shorthand for noindex, nofollow, delivered as a header rather than a tag.

The header is the correct instrument. Note the tension it now sits in: by the paragraph quoted above, a URL still disallowed in robots.txt is a URL whose headers Google is not supposed to go and read. Removal requests and the ageing of already-cached entries are separate levers we cannot observe from outside, and Google's results were already clearing before either file could have mattered. We are reporting what the two files say today, not predicting where they land. They landed two days later. The table at the top of this page shows the disallow gone by 30 July.

Now run the same mechanic in the direction you actually care about

Every business we audit eventually asks the inverse question: if I block the AI crawlers in robots.txt, am I out of AI answers? If I allow them, am I in? Our own corpus says no on both counts. The figures below come from our production audit database, cut 28 July 2026: 1,042 completed engine answers across 176 audits in four local verticals, of which 520 are ChatGPT answers. An answer counts once per domain no matter how many times it links it, so the unit is answers-citing-domain, not links. Both robots.txt files were re-fetched as plain text the same morning.

• Yelp gives OAI-SearchBot a Disallow: /, with narrow exceptions for /spk/ and /article. ChatGPT cited yelp.com in 0 of 520 answers — the finding we took apart in our Yelp piece.
• MapQuest carries an explicit, dated “Block AI/LLM Crawlers and Training Bots — Last Updated: January 2026” section handing Disallow: / to GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. ChatGPT cited mapquest.com in 12 of 520 answers, 20 links: 8 in med spas, 2 in dentists, 1 in personal-injury law and 1 in home services.

Recount, 1 October 2026. The 28 July figures reproduce exactly from a fresh export of the same database. The corpus has since grown to 1,084 completed answers across 183 audits, 541 of them from ChatGPT. Yelp is cited in 0 of 541. MapQuest is cited in 12 of 541, and the newest answer citing it is dated 11 July, so none of the 21 ChatGPT answers added since the first cut cites either site. Both robots.txt files were fetched again on 1 October and carry the same rules for OAI-SearchBot as in July. One correction: this post first listed three verticals for the 12 MapQuest answers. The cut has four, and the bullet above now lists them.

Same directive, opposite outcomes — which is the entire point. A link can reach an answer through a licensing arrangement, through the search index the engine sits on top of, or through a fetch the user triggered, and none of those is the training crawler your robots.txt is addressing. Blocking a crawler is a statement about crawling and training. It is not a statement about citation, and it has never been enforceable as one. That asymmetry is the same one Cloudflare quantified at 50,000-to-1 when it measured crawls against referrals.

Three things to do with this

1. 1To get something out of search: serve noindex (meta tag or X-Robots-Tag) and allow the crawler to fetch the page. Adding a robots.txt Disallow on top does the opposite of what you intend — Anthropic's weekend is the largest available demonstration, and Anthropic removed its own disallow within two days.
2. 2To get into AI answers: treat robots.txt as table stakes, not a lever. Don't block the retrieval agents (OAI-SearchBot, PerplexityBot) if you want to be cited. Blocking the training crawlers (GPTBot, ClaudeBot, CCBot) is a separate and defensible choice, and MapQuest is the standing evidence that it does not automatically cost you citations.
3. 3Either way, stop inferring and measure the output. No configuration file tells you what an engine cites. Only the engine does, and the answer differs by engine and by question.

If you want to know which side of this you are on, run the free check — we ask ChatGPT and Perplexity the questions your customers actually ask and show you, verbatim, whether you were named and what got cited instead.