For a publisher who wanted out of artificial-intelligence training, the past three years have required learning a separate vocabulary for each lab. Block GPTBot to refuse OpenAI. Block Google-Extended to refuse Google. Block ClaudeBot to refuse Anthropic. Add the noarchive and nocache tags to keep Microsoft’s index from storing a copy.
On Friday, four of those companies said a single line would do.
Anthropic, Google, Microsoft and OpenAI each posted support for one shared signal that site owners can place in the robots.txt file, the text document that has told crawlers what they may fetch since the mid-1990s. The proposal came from Kevin Kelley, an Anthropic engineer, as an Internet-Draft submitted to the standards body that governs web protocols. It defines a policy token, __usage_ai-training, followed by a list of paths a site does not want used to train generative models. A publisher can exempt a press kit, or a section it sells under license, in the same block.
The timing was coordinated. All four announcements landed the same day. The mechanics are not.
Anthropic said it has begun building support and warned that adding it across its services would take time. OpenAI said it aims to implement the signal “in the coming months,” while keeping its existing controls in place. Google pointed to early 2027. Microsoft set February 1, 2027, as the date its crawler begins honoring the token, replacing the archive tags it has used since 2023.
The gap between announcement and enforcement is the first thing publishers noticed. A signal that takes effect in fourteen months does not change what a crawler does this quarter, and the companies did not commit to retroactive deletion of material already ingested.
“We believe publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work,” Krishna Madhavan, a principal product manager for Microsoft’s search business, wrote in a post explaining the change. Microsoft also said the new preferences “do not establish licensing or payment arrangements,” a sentence that separates the mechanism from the money question.
That distinction carries most of the weight. A preference expressed in robots.txt is a request, not a contract. The draft states plainly that the token is not an instruction about crawling, that a site which omits it has expressed nothing, and that a private agreement between a publisher and a model developer can override whatever the file says. The document is an individual submission. The standards body has not endorsed it and it has no standing in the formal process.
Google attached a default of its own. If a site opts out with the token but says nothing about Google-Extended, the company said it will treat the silence as a block on Google-Extended. Sites that permit Google-Extended can appear in answers inside the Gemini app, where Google says they receive clicks. John Mueller, a search advocate at the company, described the arrangement on social media.
Microsoft went further on one point. It is adding a page-level setting that governs grounding, the process by which an AI answer draws on a live web page rather than stored training data. A publisher can permit Microsoft’s own products, such as Copilot, while blocking the grounding services the company sells to other firms. Blocking grounding, Microsoft cautioned, “will significantly reduce your content’s visibility in AI-generated experiences,” though it will not change where a page ranks in ordinary Bing results.
Meta’s communications chief, Andy Stone, welcomed a common signal in a post on X but did not say whether his company would honor it. Meta is not among the four signatories.
The proposal arrives with regulatory cover. Google described it as consistent with the European Union’s code of practice for general-purpose AI under the AI Act, which names robots.txt as a primary channel for machine-readable preferences. That gives the token a use beyond its technical function: a publisher that has registered a refusal in a format Brussels recognizes has a cleaner argument in a dispute than one that sent an email.
Publishers have tried this route before. A rival effort announced last year, called RSL, proposed embedding licensing terms and prices in the same file, an attempt to turn a refusal into a market. The new draft does the opposite. It standardizes the “no” and leaves the “how much” to negotiation, which is where the advantage has always sat with the party that holds the corpus.
The test will be boring and empirical. Crawlers either respect the token or they do not, and the evidence will show up in server logs over the next two years. What the four companies have agreed to, for now, is a shared format for a request each of them was already free to ignore. The value of a common vocabulary is that ignoring it becomes visible.
