The AI Training Opt-Out proposal now has backing from four of the biggest developers of generative AI. On October 30, 2026, OpenAI, Google, Microsoft and Anthropic publicly supported a common way for website owners to say whether their content may be used to train generative models. Instead of maintaining a different rule for every company’s crawler, publishers would be able to express a shared preference in the familiar robots.txt file.
The development responds to a practical problem for newsrooms, independent publishers and businesses: controlling how automated services use publicly accessible pages has become complicated as the number of AI providers grows. But this is not an instant, universal block on AI scraping. The proposal concerns the use of material for training, companies will implement it on different schedules, and the technical document is still an Internet-Draft rather than an approved standard.
What the new robots.txt token actually does
The technical mechanism is described in a proposal filed with the Internet Engineering Task Force by Kevin Kelley, whose affiliation is listed as Anthropic. Its name, Machine-Readable Training Preference Signal, sounds complicated; the underlying change is modest. It introduces a common policy token, __usage_ai-training, without changing the established robots.txt syntax or inventing new directives.
A publisher can put this example in a site’s robots.txt file:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That declares a preference against using the site’s pages for generative AI training, while permitting that use for material in the /press-kit/ directory. A sitewide preference without exceptions would use the same token followed by Disallow: /. Standard path matching rules still apply.
The distinction from a conventional crawler block matters. The draft explicitly treats the token as a statement about what consumers may do with content, not as a command that prevents a robot from fetching it. A company could still retrieve a page for another permitted purpose. A general User-Agent: * group also does not substitute for the purpose-specific training token, so administrators should not assume their current wildcard rules express the new preference.
When OpenAI, Google, Microsoft and Anthropic plan to support it
The commitments share a goal but not an activation date. In its October 30 announcement, OpenAI said it aims to honor the common signal in the coming months and will provide implementation details later. The company described participation as voluntary, making clear that support is a policy choice rather than a newly enacted legal requirement.
Google’s Search Central announcement targets support in 2027. Google says its existing Google-Extended control will remain available. Where a publisher opts out through the new shared token but does not explicitly configure Google-Extended, Google plans to interpret the shared opt-out as a Google-Extended disallow. Publishers can still express company-specific choices.
Microsoft gives the most specific date. According to its Bing publisher guidance, it will begin honoring the new preferences as pages are crawled from February 1, 2027. Microsoft’s update also adds page-level training controls and separate settings for AI grounding. Anthropic has already begun implementing support, according to industry reporting, although the rollout across its services is not described as complete.
This staggered timetable is an immediate operational limitation. A publisher adding the common rule now should not assume every participating provider is already processing it. Existing, documented crawler-specific controls still deserve attention during the transition.
AI training is not the same as AI search grounding
Training changes the learned parameters of a generative model. Grounding is different: it uses current web material to help a system answer a question, often with citations or links. A website may oppose its articles becoming training data while still wanting its reporting cited when an AI assistant responds to a reader. The new shared token deals with training, not every possible use of web content.
Microsoft’s new framework makes that distinction explicit. Its grounding controls let publishers allow use, prevent use, or choose DisallowExtended to permit Microsoft-operated experiences while restricting certain grounding services supplied to third parties. Microsoft says these choices do not change eligibility for ordinary Bing search results, subject to the usual crawl and indexing requirements. It also cautions that restricting grounding can reduce a publisher’s visibility within AI-generated answers.
Google likewise says its AI-training control can coexist with other publisher choices. For site owners, the question is therefore broader than whether to allow a bot: it is which specific uses of content support the publication’s licensing, audience and discovery strategy.
What website owners should review before editing robots.txt
The first step is to inventory existing controls instead of replacing them blindly. A technical review should identify which rules address search indexing, which mention individual AI-related crawlers, and which express training preferences. Our technical SEO audit checklist for WordPress covers the wider importance of checking crawler directives and crawl accessibility before making changes.
Administrators should then decide whether the policy applies to every page or only sections such as original reporting, paid resources or documentation. The IETF example shows that paths can be treated differently. On a WordPress installation, a virtual or plugin-generated robots.txt file may require a different editing workflow from a physical file at the site root. Whatever tool is used, the published result should be inspected directly so a typo or conflicting directive does not go unnoticed.
It is also worth separating policy signals from technical bot management. CDN, firewall and rate-limiting settings govern access and traffic behavior; training preferences declare how data should be used by participating organizations. They solve different problems. Site owners comparing those layers can consult our overview of the Cloudflare Free Plan and its limits before assuming a content-use preference functions like a security rule.
For publishers currently relying on Microsoft’s older NoArchive or NoCache settings, there is a more concrete deadline. Microsoft says the new training and grounding preferences begin taking effect on February 1, 2027, and its older directives will cease counting as those preferences after March 1, 2027. Websites that need continuity should review Microsoft’s transition instructions, particularly where grounding restrictions depend on page-level tags and Bing Webmaster Tools.
A useful signal, but not an enforcement mechanism
There are important boundaries to the announcement. The IETF’s public draft record says the document is an individual Internet-Draft, has no formal standing in the standards process, and may be changed or replaced. Its listed expiry date is May 3, 2027. Support from major companies therefore should not be confused with formal standards approval.
Nor does publishing a preference guarantee that every web crawler will comply. The mechanism depends on providers choosing to recognize and honor the signal; it does not physically prevent requests or retroactively remove content from existing model training. The Verge’s October 30 reporting also noted that Meta expressed support for a common approach without confirming that it would honor this particular token.
The wider business question is whether common preferences can reduce negotiations and technical maintenance for smaller publishers without weakening their visibility. A shared signal removes some repetitive configuration, but choices about licensing, attribution, access and search exposure remain separate. The immediate takeaway for website operators is to document their desired policy, check existing crawler controls, and follow each provider’s rollout before treating the new token as fully effective.
