OpenAI, Google, Microsoft and Anthropic back a shared robots.txt opt-out for AI training
On 30 October 2026 four AI companies announced support for a cross-vendor robots.txt training-preference signal: a site writes one group of rules under User-Agent: __usage_ai-training to tell all participating providers whether its content may be used for AI training, instead of blocking each crawler separately. Anthropic has adopted it, OpenAI says “coming months,” and Google and Microsoft target 2027. It is currently an individual IETF draft, not a formal standard.
The Verge reported on 30 October 2026 that OpenAI, Google, Microsoft and Anthropic say they will adopt a new shared signal that lets publishers state their AI-training preference to several AI companies at once, rather than configuring each company’s crawler separately. According to the report, the four companies say they will restrict their AI crawlers from accessing a site when they encounter the shared token. Meta communications head Andy Stone voiced support but did not say whether Meta will honor it.
The signal comes from an IETF Internet-Draft submitted on 30 October by Kevin Kelley, a technical lead for standards at Anthropic. Site operators write ordinary robots.txt groups keyed by the policy token User-Agent: __usage_ai-training to express, per path, whether AI training is permitted; for example Disallow: / opts the whole site out while Allow can re-enable specific folders. The draft adds no new directives or grammar, and wildcard User-Agent: * groups must not be treated as matching the token. The draft page notes it is an individual submission that is not endorsed by the IETF and has no formal standing in the standards process.
In its Search Central blog, Google says that when a site opts out of training with the token and does not explicitly specify Google-Extended, Google plans to treat that as disallowing Google-Extended. Company-specific controls such as Google-Extended keep working alongside the shared signal, and Google aims to support the control with other AI providers in 2027. Google says the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the AI Act, which recognizes robots.txt as a primary way to express machine-readable preferences.
Timelines differ. Per The Verge, Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the “coming months.” Microsoft also says it will offer more granular controls so publishers can opt out of AI training or AI grounding (pointing a model to verified sources to reduce hallucinations) separately, and that a site’s appearance in Bing search results is unaffected by these preferences.
Sources
- Supporting a shared AI training control for websites — Google Search Central Blog
- Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) — IETF Datatracker
- The biggest names in AI are backing a new way to opt out of scraping — The Verge