Skip to content
Wed 18 Nov 2026 Toronto Sign in
Software

One robots.txt rule to refuse AI training, and when it will count

An Anthropic engineer's IETF draft gives site owners a single __usage_ai-training token that Google, OpenAI and Microsoft say they will follow. Microsoft starts Feb. 1, 2027, Google says early 2027, and until then your old blocks still matter.

Illustration for the prompt/power story: One robots.txt rule to refuse AI training, and when it will count
Illustration: prompt/power

The example at the centre of the proposal is three lines long. It tells AI companies to keep everything on a site out of model training except one folder, /press-kit/, which is the part of any site that exists to be copied.

Abstract illustration: a stack of faint horizontal bars, with one longer bar in the middle lit in mint green inside a glowing rounded outline, on a dark teal gradient.
One line among many: the shared opt-out is just one group in an ordinary robots.txt file. Illustration: prompt/power

Those lines come from an Internet-Draft that Kevin Kelley of Anthropic posted to the IETF’s ai-control mailing list on Friday, Oct. 30. The same day, Google and Microsoft said they plan to support it, and OpenAI has said the same. It is easy to read that as a rule four companies now honour. It isn’t. As of Nov. 12, none of the four says support is complete, and the earliest firm date anyone has published is Microsoft’s: Feb. 1, 2027.

Real agreement, future compliance.

A user agent that isn’t a bot

The trick is in what the draft leaves out. It invents no new robots.txt command, which every parser on the web would then have to learn. Instead it defines a “policy token,” __usage_ai-training, that goes in the User-Agent line where a crawler’s name normally sits. No crawler is called that. Any parser built to the robots.txt standard, RFC 9309, can already read a group addressed to a name, and the draft says it adds no new directive names or grammar and leaves path matching untouched.

Per the draft, Disallow on a path expresses a preference against AI training for that path; Allow expresses a preference in favour. Matching is exact and case-insensitive. The double-underscore prefix is there because some deployed parsers match bot names loosely.

Then comes the clause that will catch people. A User-Agent: * group must not be treated as matching the token, and a file with no __usage_ai-training group expresses no preference at all. The blanket block you wrote for every crawler in 2023 does not count.

Nor is the token a crawling directive. The draft keeps usage preferences and traffic control separate: a company may skip fetching pages you’ve opted out, but if you want bots off your server, you still need groups keyed to their actual names.

“AI training” is borrowed from the IETF working group’s vocabulary draft, written by Paul Keller of Open Future and Martin Thomson of Mozilla, its editor, which defines it as “using an asset to modify the learned parameters of a generative AI model.” Non-generative models fall outside it. So does everything after training: search, and the retrieval that lets a chatbot quote your page live. The vocabulary has categories for AI use and search, but Kelley’s draft only defines a token for training.

Four backers, four timelines

No announcement put the timelines side by side. Here they are, from each company’s own pages where those pages would load.

Start with Anthropic, whose engineer wrote the draft. Kelley wrote on the IETF list that “Anthropic has already started implementing support for this draft,” and also that “It’ll take us some time to add support across our services.” Anthropic’s crawler help page now describes the signal, says it will roll out over time, and says rules that name ClaudeBot take priority because they are more specific. It gives no completion date.

Google offers a year in one place and a season in another. John Mueller’s Search Central post says Google aims to support the control “along with other AI providers in 2027.” Its crawler documentation narrows that to early 2027. If a site uses the shared token without naming Google-Extended, “Google plans to interpret that as if the publisher had disallowed Google-Extended.” Google also says the proposal is consistent with the EU’s Code of Practice for general-purpose AI model providers.

OpenAI is the hardest to pin down. Its announcement page would not load for us. The Next Web reports that OpenAI aims to implement the signal “in the coming months” and that existing controls such as GPTBot still apply. OpenAI’s crawler documentation did not mention the token when we checked on Nov. 12.

Microsoft gave the most detail of the four. Krishna Madhavan of Bing wrote that “Starting February 1, 2027, Microsoft will honor the new preferences as it crawls,” and that they replace the NoArchive and NoCache tags it had used for these purposes. It adds page-level control too: a Bingbot meta tag or X-Robots-Tag header with the value ai-training:disallow. Separately, it introduced a page-only grounding setting with three values. One of them, DisallowExtended, keeps a page available to Microsoft’s own AI products while excluding third-party grounding services.

“They do not establish licensing or payment arrangements.” Microsoft, on its new training and grounding preferences

Missing from the list: Meta. Its communications chief Andy Stone voiced support for a common signal on X but did not say whether Meta would honour this one, according to The Next Web, citing The Verge.

The document’s standing matters too. It is an individual submission, not a working-group draft; its own header says the IETF has not endorsed it and it has no formal standing in the standards process. The working group has a separate attachment draft by Gary Illyes of Google and Thomson, which Coywolf notes pairs a robots.txt rule with a Content-Usage HTTP header. Cloudflare, meanwhile, has shipped its own syntax since September 2025: a line like Content-Signal: search=yes, ai-train=no. Our read: the shared token wins on one count only, which is that the companies doing the training have put their names to it.

Paste this, then keep your old lines

Here is that three-line press-kit example, with the per-company names in force today stacked beneath it:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

User-Agent: GPTBot
Disallow: /

User-Agent: ClaudeBot
Disallow: /

User-Agent: Google-Extended
Disallow: /

Adjust the last three if you want any of those companies treated differently. The GPTBot and ClaudeBot blocks also stop those crawlers from fetching; the shared group alone does not. Google-Extended is a usage token rather than a separate crawler, so blocking it does not stop Google from fetching your pages.

What to settle now

1. Decide by folder, not by site. The press-kit pattern exists because most publishers want some pages copied: product docs, announcements, anything you’d like an assistant to repeat accurately.

2. Add the __usage_ai-training group now. It costs nothing, and Anthropic says it has started implementing support.

3. Keep your GPTBot, ClaudeBot and Google-Extended rules, and make them agree with the shared group. Until OpenAI and Google switch on support, their named tokens are the controls they say they read. The draft also lets companies treat their own tokens as more specific than the shared one, and Anthropic says it does. So allowing GPTBot while disallowing the shared token sends two answers, and the named bot’s line is likely to win.

4. Make the Google-Extended call yourself. Google’s crawler docs describe it as covering Gemini grounding as well as training, and the Search Central post says sites that specifically allow it remain eligible to appear in Gemini Apps responses. Use the shared token without naming Google-Extended and Google says it plans to read it as a block.

5. On Bing, move off NoArchive and NoCache. Microsoft keeps reading them until Feb. 1, 2027, says sites relying on them should finish switching by March 1, 2027, and warns that choosing Disallow or DisallowExtended for grounding means reduced brand visibility in AI experiences. It says the preferences do not change eligibility for traditional Bing search results.

6. Treat it as a request. The signal is voluntary, and private agreements can override it. Bots that ignore robots.txt altogether have to be blocked at the network edge.

One more date for the calendar. The draft expires on May 3, 2027, about three months after Microsoft starts reading it. Unless Kelley posts a revision or the working group adopts it, the biggest crawlers on the web will be following a document that, by IETF convention, is cited only as “work in progress.”

Abstract illustration: a single mint line runs from a small sphere to a fork, splitting into two curved paths that each end at a closed vertical bar, with a faint dashed bypass looping around the lower bar to a lime dot, on a dark teal gradient.
At Google, one opt-out can close two doors, training and grounding, unless a separate Google-Extended allow reopens the second. This is our read of Google’s documentation. Illustration: prompt/power
// How we reported this

Researched and drafted with AI tools, fact-checked against primary sources and approved by a human editor before publishing. Prices and dates are as of publication. Our standards →

Sources
  1. datatracker.ietf.org — an Internet-Draft
  2. mailarchive.ietf.org — posted to the IETF's ai-control mailing list
  3. developers.google.com — Google
  4. blogs.bing.com — Microsoft
  5. openai.com — OpenAI
  6. rfc-editor.org — RFC 9309
Columnist, Creator Economy & Culture Casie Stewart

Casie Stewart covers the creator economy, social platforms and wellness tech for prompt/power: who gets paid, who gets seen, and what the algorithm is really asking of the people who feed it. She has been publishing online since 2005, long before "creator" was a job title, and has spent most of that time on the other side of the platforms she now writes about. Based in Toronto. Her rule for any new app: if it promises to save you time, check how much of it you just spent reading the onboarding.

All stories by Casie →

Latest from prompt/power

  1. Nvidia beats its own forecast with US$115.8 billion, then guides margins lowerNov 18
  2. Apple, the NSA and Chase: what each says broke in a week of outagesNov 18
  3. ChatGPT for Healthcare now has agents that click through hospital software. The fine print is all AmericanNov 17
  4. ChatGPT Wallet lets an AI agent pay for you. Here’s who can use it, and why Canada and the U.K. waitNov 17
  5. AntiFox: attackers are using an AI pentest agent to steal AWS secrets. Six checks to runNov 17

Leave a Reply

Your email address will not be published. Required fields are marked *