Does robots.txt stop ChatGPT from fetching your pages? OpenAI says ChatGPT-User rules "may not apply" — and TollBit measured a 54% bypass rate

What TollBit measured

TollBit sells bot-paywall and licensing infrastructure to publishers; its half-yearly State of the Bots reads the first-party server logs of the publisher sites on its network — self-identified AI user agents only, so every figure below is a floor. The H1 2026 edition ("The Bad Bots") splits European from North American sites; Section 4 is the robots.txt half. Its definitions are blunt: a disallow is a robots.txt instruction naming a specific user agent; a bypass is "when an AI bot scrapes a publisher's content despite that publisher having explicitly disallowed that bot" — any request to a disallowed URL counts, and the report notes AI developers say a new disallow can take 24–72 hours to be reflected. Search Engine Journal carried the ChatGPT-User finding on 2026-08-14; the numbers below are read from the report itself.

Two qualifications belong next to the headline. First, the bypass rate is measured against sites that explicitly disallowed the bot, and the report notes ChatGPT-User is disallowed "at a rate closer to the established crawlers, most likely because it is the most widely recognized" — it has more disallows to bypass than Claude-User or Perplexity-User do. Second, this is a publisher cohort: news, sports, entertainment. It says what OpenAI's fetcher does to a newspaper that said no; it does not say what a business that wants to be found should do. That is the part we can add.

What OpenAI's own documentation says

OpenAI's Overview of OpenAI Crawlers, read today, describes three user agents that matter here and assigns each a different job. OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features" — "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." GPTBot "is used to crawl content that may be used in training." And ChatGPT-User: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent … ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl." The doc also notes robots.txt changes take "~24 hours" to propagate for search. So TollBit's 54% is not a broken promise — OpenAI never made one for ChatGPT-User. The dispute is whether a user asking a chatbot to read a page is the user fetching it or the bot; publishers, per TollBit's foreword, say the bot.

Our cut: the domains AI engines actually cite do not block ChatGPT-User — none of them

TollBit's population is publishers who told bots no. The population that matters for AI visibility is the domains engines cite. So we took the 2026-08-10 run of our weekly self-audit — 12 pinned small-business buyer questions × 4 engines × 3 samples = 144 production answers, 540 distinct cited domains — ranked domains by how many answers cited them, and fetched robots.txt for the top 150 today with the same guarded parser we used in our July rule-parse (a file counts as measured only on HTTP 200 + `text/plain` + no `<html` in the body; anything else is UNMEASURED and never reported as a block). 142 measured, 8 unmeasured (linkedin.com and aivisitaudit.com serve HTML; three return 202; one 404; one 403; one no DNS). Ten agents parsed per domain: ChatGPT-User, OAI-SearchBot, GPTBot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, CCBot.

Every one of the 16 path-scoped ChatGPT-User entries is housekeeping — `/api/`, `/admin/`, `/dashboard/`, `/login`, `/_next/` — under an explicit `Allow: /`; not one fences off content. 90 of the 142 name no AI agent at all. Among the 40 measured domains ChatGPT itself cited, zero block GPTBot, ChatGPT-User or OAI-SearchBot. The only site-wide `Disallow: /` under `User-agent: ` in the set is reddit.com, which four engines cite anyway. And forbes.com fully disallows PerplexityBot yet was cited by Perplexity in 8 of the 144 answers — the same naming-is-not-blocking, blocking-is-not-invisibility pattern we found in July, now on a population 7× larger. To be exact about what this is: a selection effect, not a treatment effect. We cannot say blocking causes* a site to drop out of answers; we can say that in the set of domains that make it into answers on real buyer questions, blocking the fetchers is nearly absent (10 full blocks across 1,278 measured domain-agent pairs), and blocking ChatGPT-User specifically is absent entirely.

Why this matters if you care about AI visibility

• Three lines, three different outcomes — get the one you mean. Per OpenAI's doc, `OAI-SearchBot` decides whether you can appear in ChatGPT search answers, `GPTBot` decides training, and `ChatGPT-User` is the fetch that happens when a user asks about a page — the one OpenAI says robots.txt may not govern, and the one TollBit measured bypassing 54% of the time. A business that wants to be cited has exactly one line to protect (allow OAI-SearchBot) and nothing to gain from disallowing ChatGPT-User, which neither stops the fetch reliably nor affects search eligibility.
• Your ChatGPT-User log lines are a demand signal you can trust. TollBit's IP check found under 2% of scrapes claiming an OpenAI, Anthropic, Amazon or Perplexity agent came from outside that platform's published ranges (ChatGPT-User: 0.65%) — "if you are seeing these AI platforms accessing your site based on user agents, it is highly likely that the traffic is actually coming from the relevant platform." Each ChatGPT-User hit is a person who asked ChatGPT something that made it read your page. That is measurable at the server, which is why Microsoft Clarity's new per-operator scrape-to-referral ratio needs a CDN or server log integration — and why the referral column understates it: only ~1% of AI-shaped visits arrive with an AI referrer.
• The exchange rate is terrible and the answer presence is real — those are two different facts. 179 RAG scrapes per human referral in Europe, 265:1 for OpenAI, AI apps at 0.05% of external referrals — TollBit's numbers sit beside Cloudflare's 50,000:1 and are just as bleak. A publisher whose product is the page can rationally block. A business whose product is the thing being asked about is playing for the mention, and every one of the 142 most-cited domains on our questions is playing that way. The number that tells you whether it is working is not in any log: ask the engines about your business the way a customer would, and see who they name.