On Oct. 30, Kevin Kelley, a technical lead for standards at Anthropic, posted a new draft to the mailing list of the Internet Engineering Task Force's AI Preferences working group. Its working part fits in two lines of a robots.txt file: a group headed User-Agent: __usage_ai-training, followed by Disallow: /. Together they mean "don't use this site to train generative AI models." The same day, Google, Microsoft and OpenAI each said they would honor it.
For site owners, that removes a chore. Until now, OpenAI wrote in its announcement, operators had to "identify individual crawlers and maintain separate rules for each and every provider." A shared token lets one line speak to every company that signs on.
What the companies agreed on, though, is the meaning of one word: training. The part most publishers actually weigh, whether their pages still show up and get clicks inside AI answers, is still set company by company. Under the rules Google and Microsoft have published, the same two lines touch only training at Microsoft, but at Google they also cost a site its eligibility to be cited in the Gemini app. The way to opt out is now shared. What opting out costs is not.
Same line, different results
Google's post says that when a site uses the new token to refuse training and doesn't separately address Google-Extended, Google will treat it as if the site had disallowed Google-Extended. Google's crawler documentation says Google-Extended governs more than training. It decides whether crawled content can train Gemini models and whether it can be used for grounding, meaning content pulled in at answer time, in Gemini Apps and in Grounding with Google Search on Vertex AI. Google's post adds that sites that specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps."
So a small site that only wants to keep its work out of training, and writes just those two lines, also drops out of Gemini's cited answers, going by Google's own documents. Keeping that exposure means adding a rule that allows Google-Extended, which allows training too. AI Overviews and AI Mode in Google Search sit behind yet another switch in Search Console, and Google's help page says that control "doesn't affect AI training."
Microsoft went the other way and pulled the two apart. A post by Krishna Madhavan, a principal product manager for Bing, says the shared token covers training only. Microsoft is adding a separate AI Grounding preference that can be set only page by page, with three values: allow, disallow, and a new DisallowExtended, which keeps content in Microsoft's own products such as Copilot and Microsoft 365 but out of grounding services it sells to others, such as Web IQ and the grounding API in Foundry. There is no site-wide grounding setting in robots.txt.
Defaults and deadlines
Both regimes share one assumption: silence means yes. The draft says wildcard groups such as User-Agent: * must not be read as matching the new token. Kelley wrote that this was deliberate, so a robots.txt file that doesn't use the mechanism, "as is true of every one published today, expresses no preference at all." Microsoft says both training and grounding default to allow when nothing is set.
That suits most of the web. Sites that are happy to be used change nothing, and existing crawler rules keep their meaning. The cost lands on sites that want out: they have to act, and they have to learn what the choice means at each company.
Microsoft adds a clock. It will start honoring the new preferences as it crawls on Feb. 1, 2027. Sites that relied on its older NoArchive and NoCache tags to limit AI use have until March 1, 2027, to publish the new page-level grounding preference and record a temporary setting in Bing Webmaster Tools. Pages that skip the second step default to allowing grounding until Bing recrawls them. After March 1, the old tags no longer count as a training or grounding preference.
Opting out isn't blocking
The draft draws another line that is easy to miss. The token is about how content is used, not whether it is fetched. Kelley called the two "intentionally separate": a crawler's own product-token rules still govern crawling, and the draft only asks crawlers not to fetch what they can't use. Microsoft says the same, and notes that a new setting takes effect only after it crawls the page and sees it.
Timing varies too. Kelley said Anthropic has started implementing the draft but needs time to cover all its services. OpenAI says it will implement it "in the coming months." Google's post says 2027; Microsoft names Feb. 1. Meta's communications head, Andy Stone, backed the idea of a common signal, The Verge reported, without saying whether Meta would honor it. The document itself is an individual Internet-Draft. Its Datatracker page says it is "not endorsed by the IETF" and has no formal standing in the standards process. It expires May 3, 2027.
The other route: close the door
A day earlier, Reddit used its third-quarter earnings call to announce a gradual shift to a mostly logged-in site. TechCrunch reported that the move is meant partly to shield its data from AI scraping so it can strike licensing deals. Reddit said search referrals fell by more than 1 million a week on average in October. Its non-advertising revenue, which includes AI data licensing, rose 35% from a year earlier to $48 million. Chief Executive Steve Huffman wrote that as "someone whose life has been defined by the open internet, it's a big change for me," but that it was "the right decision for our platform and the right decision for our business."
Reddit is big enough to lock its content up and negotiate. Most independent blogs, open-source documentation sites and small product teams aren't, and robots.txt is the tool they have. For them, a shared training token does make it cheaper to state a position.
But the choice many small sites actually face isn't "train on me or don't." It's "cite me, but don't train on me." At Microsoft, that combination can be set page by page. In the Gemini app, under Google's current documents, the two still ride on the single Google-Extended switch. Microsoft's own post cautions that no setting "guarantees inclusion, traffic, or payment."
The working group still has to define the uses beyond training. Until it does, those two lines settle the meaning of one word, and site owners still have to read each company's documentation before they use them.