Google, OpenAI, Microsoft and Anthropic have each said they will honor one shared robots.txt line for opting a site out of AI training, The Verge's Emma Roth reported. It is a proposal, not a finished standard, and only Anthropic has said it has started, in part.
What to do: add the token to robots.txt
Add the block below if you want to be covered as support arrives. It does nothing yet at most of these companies. The company-specific rules you already have, such as Google-Extended, remain what works today, so keep them.
The signal is a policy token, __usage_ai-training, used where a crawler name normally goes. The example in the IETF draft is:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That says the site prefers no AI training on any path except /press-kit/. The token must match exactly, apart from letter case. A User-Agent: * group does not count, and if no group names the token, the site has expressed no preference.
Who honors it, and when
- Microsoft: begins honoring the preferences on Feb. 1, 2027, as it crawls. Microsoft's post says a page's setting updates only when it is recrawled. A page can also carry a Bingbot meta tag,
ai-training:disallow, or an X-Robots-Tag header, which overrides robots.txt. - Google: says it plans to support the token and aims to do so, along with other AI providers, in 2027. Google's post says that if a site opts out with the token and does not explicitly set Google-Extended, Google plans to treat it as if Google-Extended were disallowed.
- Anthropic: the draft's author, Kevin Kelley, is at Anthropic. In a post to the IETF mailing list he said Anthropic has started implementing support, that rollout across its services will take time, and that its interpretation may change as standards evolve. Anthropic says the signal adds to its existing controls, not replaces them.
- OpenAI: its post says OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers. OpenAI's own post gives no date, but OpenAI will honor it in the coming months, according to the same report.
What it does not do
- It is not a crawl rule. It covers generative-AI training only. Microsoft says opting out of training does not affect eligibility for regular Bing results.
- It is not a finished standard. The draft is an individual submission, not endorsed by the IETF, and expires May 3, 2027.
Why it matters
One entry replacing a list of per-company rules would save site owners work, but the cost is a wait. Until 2027 the token does little. Microsoft says the new preferences replace its older NoArchive and NoCache tags; sites using those should set the new preferences by March 1, 2027, after which the old tags are no longer read as a training or grounding preference.
Related
This follows Meta's Commercial Terms crawling permission for business websites. For sites shutting AI agents out entirely, see how banks and Yelp are blocking AI agents. More is in AI & Apps.
