For years, telling AI companies not to train on your work meant writing a separate rule for every company, then updating the list every time a new crawler appeared. On October 30, 2026, the four biggest AI developers agreed to end that chore — with one shared line in a file every website already has.
Google, Microsoft, OpenAI, and Anthropic said they will honor a single robots.txt signal for AI training opt-outs, built around the token __usage_ai-training. Meta expressed support without committing to a date. For creators, publishers, and freelancers who sell content, this is the closest thing yet to a standard “do not train” switch.
It is also not what the headlines suggest. Nobody honors the line yet, it does not stop crawling, it does not undo past training, and it can cost you visibility in AI answers. The money is in helping clients use it correctly — and in selling what the switch alone cannot give them: a license.
What Is Actually Trending
Robots.txt is a small text file at the root of a website that tells automated visitors which pages they may fetch. Until now, opting out of AI training meant naming each company’s crawler: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and more — a list that grows with every new lab.
The new approach names a purpose, not a company. A site adds a group naming the shared token, then a Disallow rule, like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That example blocks training use of the whole site while leaving a press-kit folder open. The draft was written by Kevin Kelley, Anthropic’s technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an individual draft on October 30. It is not yet a formal internet standard, and it deliberately uses no new grammar: path matching follows RFC 9309, the robots exclusion protocol.
One trap is already documented. A wildcard group — User-Agent: * — must not be read as matching the token. A site that blocks every crawler with one wildcard rule has, under the draft, said nothing about AI training at all. The token has to be named explicitly.
The Evidence: One Pledge, Four Different Timelines
The four companies did not promise the same start date, and that gap is the story. Anthropic says it has begun implementing the signal. OpenAI aims to honor it “in the coming months.” Microsoft gave the firmest date. Google gave a target window.
Sources: Nieman Journalism Lab, “Anthropic, OpenAI, Google, and Microsoft will make it easier for publishers to opt out” (Nov 2, 2026) · Search Engine Journal, “Google, Microsoft, OpenAI plan to honor a shared AI training opt-out” (Nov 1, 2026) · Tech Insider, “AI Training Opt-Out: 4 Giants Agree on 1 Standard” (Nov 1, 2026) · MediaPavilion, “AI Training Opt-Out: One Robots.txt Signal Explained” (2026).
The need for a shared signal is not theoretical. Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a separate standard called Really Simple Licensing (RSL) already lets publishers set terms for crawlers. But enforcement of the old patchwork was weak: PPC Land cites TollBit data finding about 15% of identified AI page-fetching agents reached URLs that European sites had disallowed, and a HasData test in July found 234 of 592 sites that disallowed GPTBot still served it a live page. A signal only works against companies that choose to read it.
Why the Switch Is Worth Less Than It Looks — and More
The switch is worth less than it looks for three reasons, and each one creates a service.
It is a preference about use, not a block on crawling. The draft states the token “is not a directive for crawling behavior.” A page can be kept out of training while still being fetched for search or other purposes — and Microsoft confirmed sites will still appear in Bing search results. If a client wants a bot to stop visiting entirely, they still need to block it by name.
It covers future training only. Content already collected into datasets or trained into existing models is untouched, and robots.txt has always worked on trust: a scraper that ignores the file today will ignore this entry too. Smaller labs, open-weight builders, and overseas crawlers have made no commitment at all.
It can cost visibility. Google and Microsoft both warned that refusing training use could reduce the chances that content surfaces in AI apps and answers. For a publisher whose traffic now comes from AI answers, opting out of training may also mean opting out of being cited.
And it is worth more than it looks for one reason: bargaining power. A clean, widely honored refusal makes it easier to argue that any use without a deal was unwelcome — which strengthens the case for a paid license. Microsoft said explicitly that the new preferences “do not establish licensing or payment arrangements,” which is exactly why the license has to be sold separately. The switch says no. The license is how a client says yes, for a price.
Decide, Signal, License, Monitor
The opt-out is step two of four. Clients who stop at step two get the weakest version of the deal.
Choose per section
Not every page deserves the same rule. Open marketing pages, protect the work that earns.
Write the token correctly
Name __usage_ai-training explicitly, keep company rules in place, and test the file.
Price the yes
A licensing page or RSL terms turn a refusal into a paid permission for buyers who want the content.
Watch the crawlers
Log who visits, re-check after each company’s start date, and update rules as new crawlers appear.
How to Profit From This Trend
Every site owner with original content now has a question they cannot answer confidently: does my robots.txt actually say what I think it says? The offer is a content-rights audit: read the current file, list every AI rule (including the wildcard trap), explain what each company will honor and when, and rewrite it with the shared token plus the company rules that still matter.
The larger sale is the opt-out plus license pack: the corrected robots.txt, a public licensing page with terms and a contact for buyers, and a section-by-section decision (what stays open for discovery, what is closed for training). The retainer is licensing watch: crawler logs reviewed monthly, rules updated as companies flip their support on, and new licensing inquiries routed to the client.
Prices below are my estimates based on comparable freelance web and publishing service rates, not market data:
Lead with the wildcard trap. “Your site may already block every bot — and still have said nothing about AI training.” Show the client their own file, the missing token, and the February 1 date when Microsoft starts reading it. The audit sells the moment they realize the protection they assumed they had was never written down.
What to Do This Week
- Read your robots.txt today. Look for GPTBot, Google-Extended, ClaudeBot, and any wildcard rule. Note what is missing before you change anything.
- Add the shared token now, and keep the company rules. The token only starts counting as each company turns support on, so early placement costs nothing — but removing old rules early removes protection that still works.
- Decide section by section. Keep discovery pages open where AI visibility helps you, and close the premium work. A blanket block is the laziest decision, not the safest one.
- Write the license before you need it. A short licensing page with terms and a contact turns future inquiries into revenue instead of ignored emails.
- Calendar the start dates. February 1, 2027 for Microsoft, early 2027 for Google, and “coming months” for OpenAI are the dates to re-check your file and your crawler logs.
Watch: a short explainer on the shared opt-out signal (daily.dev). Watch on YouTube.
Where This Is Heading
The draft becomes a standard — or a footnote. The IETF process will decide whether __usage_ai-training becomes a real standard or stays a four-company handshake. Either way, the pattern is set: purpose-based signals, not crawler lists, are how the web will negotiate with AI.
Licensing splits from blocking. Publishers who only block will watch licensed competitors get paid. The winners will run both: a clear no for training, and a clear price for anyone who wants to say yes.
Visibility becomes a priced choice. As AI answers replace search clicks, the decision to opt out will be weighed against citation traffic, not principle alone. Expect tools that show a publisher exactly how much traffic each choice costs — and consultants who sell that answer.
Frequently Asked Questions
What is __usage_ai-training?
It is a shared policy token for robots.txt, proposed in an IETF individual draft by Anthropic’s Kevin Kelley on October 30, 2026. A site names the token and adds Disallow rules to express a preference against generative AI training use of its content, readable by every participating company at once.
Which companies will honor the shared opt-out?
Google, Microsoft, OpenAI, and Anthropic have backed it, and Meta has expressed support without an implementation commitment. Anthropic is already implementing it, Microsoft starts February 1, 2027, Google targets early 2027, and OpenAI aims for the coming months.
Does the token stop crawlers from visiting my site?
No. The draft states the token is not a directive for crawling behavior — it expresses a preference about training use. A page can be kept out of training while still being fetched for search or other purposes. To stop visits entirely, block the crawler by name.
Does a wildcard robots.txt rule already opt me out?
No. Under the draft, a wildcard group (User-Agent: *) must not be interpreted as matching the policy token. To opt out, the token has to be named explicitly, even if your site already blocks every crawler.
Does opting out undo past AI training on my content?
No. The signal covers future training only. It does nothing about material already collected into datasets or trained into existing models, and private licensing agreements can override the signal for a specific company.
Will opting out hurt my search or AI visibility?
Possibly. Google and Microsoft warned that refusing training use could reduce the chances content surfaces in AI apps and answers, because Google reads the shared opt-out as blocking Google-Extended, which covers Gemini grounding. Weigh visibility against protection section by section.