AI Training Opt-Out: One robots.txt Rule Explained

 

For years, telling AI companies not to train on your website meant playing whack-a-mole. You blocked one crawler by name, another company launched a new one, and your robots.txt file grew longer every few months. On October 30, 2026, the four biggest Western AI labs said they want to end that mess with a single shared signal.

OpenAI, Google, Microsoft, and Anthropic backed one AI training opt-out in robots.txt: a shared policy token called __usage_ai-training. Add it once, and — once each company has switched it on — every participating provider should read the same preference. It is the clearest joint move publishers have seen on AI training, and it is also not a switch you can flip today and forget. Support arrives on different dates, it is voluntary, and it only covers training. Here is what actually changed, what did not, and exactly what to put in your robots.txt file.

Quick answer: The new AI training opt-out is a shared robots.txt token, __usage_ai-training, backed on October 30, 2026 by OpenAI, Google, Microsoft, and Anthropic. A site that adds User-Agent: __usage_ai-training with Disallow: / states that its content should not be used for generative AI training. It does not block search crawling, does not remove content already collected, and is not live everywhere yet: Anthropic says it has started, OpenAI says “in the coming months,” Microsoft starts February 1, 2027, and Google aims for 2027.

Contents

• What was announced on October 30?
• How the shared token works
• How to add the AI training opt-out to robots.txt
• When each company will honor it
• Training is not the same as AI answers
• What this opt-out does not do
• Should you opt out, or stay in?
• Pros and cons
• FAQs
• Verdict

What was announced on October 30?

All four companies published coordinated announcements on the same day, built on work from the Internet Engineering Task Force (IETF) AI Preferences working group, known as AIPREF. The technical write-up is an individual Internet-Draft, “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00), written by Kevin Kelley, a technical lead for standards at Anthropic, and dated October 30, 2026.

That last detail matters. An individual Internet-Draft is not an approved IETF standard, and the draft itself expires on May 3, 2027 unless it is revised or adopted. The announcements are commitments to honor a shared signal as it is standardized — not proof that every crawler obeys it this week.

Before this, a publisher who wanted out of training had to know every relevant crawler by name: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and others as they appeared. Miss one name, and that company’s training crawl was not covered. The shared token replaces that list with one entry that participating companies agree to read the same way. OpenAI put the old problem plainly: website operators previously had to identify individual crawlers and maintain separate rules for each provider, and a shared signal reduces that work.

Meta, notably, did not join the commitment. Meta’s communications head welcomed a common signal as an important step, according to reporting on the announcement, but Meta did not confirm it will honor the token. If Meta’s crawlers matter to you, this announcement alone does not cover them.

The timing also tells a wider story. Governments and institutions are drawing their own lines around AI tools at the same time — Sweden blocked ChatGPT and Claude on government work computers in October 2026, keeping only Copilot. The robots.txt signal is the publishers’ version of the same question: who gets to decide how content and tools are used, and where is that choice recorded?

How the shared token works

Robots.txt has always been a plain-text file at the root of a site, like yoursite.com/robots.txt, that tells automated visitors what they may do. The new signal reuses the familiar robots.txt format — no new grammar, no new directive names — and puts the preference where a crawler name normally goes.

The token is __usage_ai-training, with two leading underscores. It is a “policy token,” not a real crawler. A participating AI provider reads the group keyed by that token as a statement about one purpose only: generative AI training.

Three rules from the draft are easy to get wrong, so they are worth stating plainly:

• It is not a crawl block. The draft says the policy token is not a directive for crawling behavior. Disallow: / under this token means “do not use this for training,” not “do not visit this site.” Search crawlers such as Googlebot use separate rules, and Google says the new control does not affect a site’s inclusion in Google Search.
• User-Agent: * does not count. A wildcard group must not be read as matching the policy token. If no group names __usage_ai-training exactly, the site has expressed no training preference at all, whatever its catch-all rules say. You have to name the token.
• Specific beats general. A company-specific setting, such as an explicit Google-Extended rule, takes precedence over the shared token for that company. That is how Google can map the shared opt-out onto Google-Extended while still letting a site explicitly allow Google-Extended if it chooses.

Google’s Search Central post, by Search Advocate John Mueller, explains the mapping: when a site uses the shared token to opt out of training and does not explicitly mention Google-Extended, Google plans to treat that as if the publisher had disallowed Google-Extended. Company-specific controls keep working alongside the new signal, and Google-Extended itself is not going away. Google has offered Google-Extended since 2023, and it has long said that control does not affect inclusion or ranking in Google Search.

You can also carve out exceptions with Allow: lines, using the same longest-match path rules as ordinary robots.txt. The example discussed in coverage of the draft opts out of everything except a press-kit folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That pattern is useful for a newsroom that wants its public press material to stay available for training while keeping the rest of its reporting out, or for a creator who is happy for free samples to circulate but not a full archive. Test any exception carefully — path matching in robots.txt rewards precision and punishes typos.

How to add the AI training opt-out to robots.txt

Adding the token now is future-proofing, not an instant block. At most companies it does nothing until their support goes live, so keep the company-specific rules that work today. Think of the shared token as the line you add alongside them, ready for 2027.

Here is the step-by-step:

1. Find your robots.txt file. It lives at the root of your domain. On WordPress, use your SEO plugin or host file manager. On a static site, edit the file in your repository or hosting dashboard. On Blogger, check Settings, then Crawlers and indexing, then Custom robots.txt — if your host controls robots.txt and does not let you edit it, you cannot add this signal directly, and no plugin can fake it.
2. Back up the current file first. Copy the whole file into a note before you touch it. One misplaced Disallow under the wrong user-agent can hide a site from search, and a backup turns a panic into a two-minute fix.
3. Add the shared group on its own, at the end of the file. Do not merge it into an existing group, and do not put it under User-Agent: *:

   User-Agent: __usage_ai-training
   Disallow: /

4. Keep your existing per-company rules for now. Keep Google-Extended, GPTBot, ClaudeBot, and any other training controls you already use. The companies say the old controls keep working alongside the new signal, and they are the only ones with live effect until each company’s start date.
5. Add exceptions only if you mean them. If part of your site should stay available for training, add an Allow: line for that path inside the same group, as in the press-kit example above. If you are unsure, leave the group as a full opt-out — a narrow exception added later is safer than a broad one added by accident.
6. Save, then check the live file. Open yoursite.com/robots.txt in a browser and confirm the new lines are there, spelled exactly — two leading underscores, one hyphen, no spaces inside the token. A single missing underscore turns the whole group into a rule nobody recognizes.
7. Re-check in 2027. The draft expires in May 2027 unless it is updated, and each company’s implementation may add detail. Put a calendar reminder for February 2027 to confirm Microsoft’s start date has held, and another for mid-2027 to confirm the token is still the current form.

If you run a site where pages need different treatment — say, free guides open for training but a paid archive closed — Microsoft is going further than the shared file-level signal. It is extending the AI training preference to the page level, so a single page can override the robots.txt preference with a meta tag or HTTP response header naming Bingbot. Microsoft’s documentation is the place to confirm the exact syntax before you use it; do not copy a page-level tag from a social post. For page-level controls in general, our Meta Tag Generator can build the surrounding meta tags for a page, but the AI training preference itself should follow Microsoft’s published format, not a generic template.

One warning for Blogger and other hosted sites: Blogger’s own robots.txt handling is separate from an individual post. Adding a line to a custom robots.txt file, if your settings allow it, applies site-wide. There is no per-post robots.txt opt-out in this announcement, so do not expect a single article to be excluded while the rest of the site stays in.

When each company will honor it

The four timelines are not the same, and that gap is the most practical fact in the whole announcement. A signal added today may be read by one company months before another.

Company	What it committed to	When
Anthropic	Honor the shared token for its crawlers	Already started implementing support
OpenAI	Honor the shared signal for foundational generative AI training; existing crawler controls stay in force	“In the coming months,” with a further update promised
Microsoft / Bing	Honor the shared training preference, plus separate grounding controls; effective as Bingbot recrawls pages	February 1, 2027
Google	Support the shared control; a shared opt-out with no explicit Google-Extended setting is read as a Google-Extended disallow	Aims for 2027; crawler documentation narrows it to early 2027
Meta	Welcomed a common signal; no commitment to honor this token reported	No date

Microsoft also published the firmest surrounding dates. Temporary grounding controls open in Bing Webmaster Tools on December 15, 2026 for eligible sites that used older NoArchive or NoCache directives. Sites relying on those older tags have until March 1, 2027 to set the new preferences; after that, Microsoft says NoArchive and NoCache will no longer be read as a training or grounding preference. Temporary grounding preferences expire on September 30, 2027. If your site ever used NoArchive or NoCache to limit AI use on Bing, that March 1 date is the one to put in your calendar — it is a real deadline, not a vague “next year.”

Until those dates arrive, nothing about this announcement changes what a crawler does today. The honest way to describe the current state is: announced and agreed in principle, implemented in part, live everywhere only in 2027 at the earliest.

Training is not the same as AI answers

The single most misunderstood part of this story is the difference between training and grounding. Training changes a model’s learned parameters: your text helps shape how the model itself behaves, long after it was collected. Grounding is different. A model fetches a live page to answer a user’s question, quotes or cites it, and may send the reader back to your site. Opting out of training does not, by itself, opt you out of grounding.

That distinction is why Microsoft built two separate preferences. Its AI Training setting controls whether content may train models. Its AI Grounding setting controls whether content may help generate AI answers in products such as Copilot, Bing AI features, and Microsoft 365. Both default to Allow when no preference is published, and Microsoft says neither choice changes eligibility for ordinary Bing results — though it also warns that neither choice guarantees inclusion, traffic, or payment.

Microsoft’s grounding control has a middle option, DisallowExtended: grounding is allowed in Microsoft’s own products, but not through grounding services Microsoft sells to other companies. In other words, you can refuse training, allow your pages to inform answers inside Copilot, and still keep them out of third-party grounding services. That granularity is new, and it is the most publisher-friendly part of Microsoft’s announcement — provided you actually set it, rather than assuming the training token covers it.

Google draws a related line with a different tool. Its Search generative AI control in Search Console, rolled out to all websites worldwide on August 31, 2026, controls whether a site’s links and content appear in AI Overviews, AI Mode, and generative features in Discover. Google is explicit that this control does not affect AI training — for training, it points publishers to Google-Extended. The new shared token is the training-side control; the Search Console setting is the answers-side control. They are complements, not substitutes, and a site that wants both limits needs to set both.

Google also links the two sides with an incentive worth reading before you decide. Sites that explicitly allow Google-Extended remain eligible to appear in Gemini Apps responses through grounding and to receive clicks from it. The shared-token opt-out, read as a Google-Extended disallow, points the other way. If cited answers and the traffic that comes with them matter to your site, that trade-off deserves a deliberate decision, not a default.

What this opt-out does not do

A fair reading of the announcements needs the limits stated as clearly as the promise:

• It does not remove content already collected. Updating robots.txt does not delete material already in existing datasets or already trained into model weights. The signal is judged against the robots.txt file in effect when a page was fetched, so it looks forward, not back. If your work is already in a training set, this token does not take it out.
• It does not block search. It is a training preference, not a crawl or indexing rule. Your pages stay in Google Search and Bing results. If a page must stay out of search entirely, that is what noindex is for — a different tool with a different cost.
• It is not a law, and it has no enforcement mechanism. Robots.txt has always worked on trust. The signal is voluntary, the draft is not an approved standard, and a company that ignores robots.txt faces no automatic penalty from the file itself. Legal disputes over training data — the kind that produced the New York Times lawsuit against OpenAI and Microsoft in December 2023 — are fought in court and in regulation, not in robots.txt.
• It does not cover every AI company. Four labs backed it, Meta did not commit, and smaller model builders, scrapers, and dataset collectors are not bound by a press announcement. A site that wants broad protection still needs its per-company rules, terms of use, and, where it matters, legal advice.
• It does not guarantee invisibility in AI answers. As Microsoft and Google both make clear, training, grounding, and search-AI features are separate choices with separate controls. One token in one file does not set all three.

Those limits do not make the announcement empty. A shared, machine-readable preference that four major labs agree to read the same way is genuinely easier to maintain than a growing list of crawler names, and it gives smaller publishers a one-line option they did not have before. But it is a preference signal, not a lock on the door — and articles that describe it as a ban are overstating it.

Should you opt out, or stay in?

There is no single right answer, and the honest answer depends on what your site is for. Work through these cases before you edit anything:

Opt out if your content is the product. Newsrooms, authors, photographers, and research publishers whose archives are the asset have the strongest case. If the value of your work is in the reporting, the analysis, or the original material itself, letting it train a model that may later answer the same questions without a visit is a real cost. Legal teams tracking AI and copyright will recognize the shape of this dispute — it is the same fight that has run through the courts for three years, now with a simpler technical expression.

Stay in — or opt out only in part — if discovery is the product. A new blog, a small business, or a creator site often gains more from being quoted, cited, and linked in AI answers than it loses from training. ChatGPT’s newer interfaces and Gemini’s grounded answers increasingly send readers somewhere; being citable is a distribution channel. For those sites, the Allow: exception is the useful middle path: keep flagship free content available, and close the archive, the paid course, or the licensed dataset.

Authors and creators should think in layers. The Character.AI bankruptcy case showed how quickly questions about user chats and platform data can become urgent when a company fails. A robots.txt token is only one layer of control over your work — alongside licenses, terms, and where you choose to publish in the first place. Do not treat it as the whole strategy.

Check the scale of the stakes with real numbers. Before deciding, look at how much of your traffic already comes from search and AI answers, and how much of your revenue depends on visits rather than reputation. Our AI statistics roundup for 2026 gives a sense of how fast AI-mediated discovery has grown; your own analytics will tell you whether your site is riding that wave or being bypassed by it. A decision made from your own data beats a decision made from a headline.

Whatever you choose, write it down. Record the date you added the token, which paths you excepted, and which company-specific rules you kept. When the draft is revised or a company changes its implementation in 2027, that note is what lets you update in minutes instead of reconstructing a decision you made months earlier.

The same “decide deliberately, then set it once” logic applies to the assistants now doing proactive work on users’ behalf. Anthropic’s Claude Duet, which reads connected email and calendars and surfaces tasks each morning, is a reminder that AI systems increasingly act on content they can reach — and that the preferences site owners express in machine-readable files are becoming part of how those systems are governed.

Pros and cons

Pros:

• One line replaces a growing list of per-company crawler rules, and it is readable by every participating provider.
• It reuses the robots.txt format site owners already know — no new file, no new syntax to learn.
• It does not hurt search inclusion or ranking, according to both Google and Microsoft.
• Exceptions are built in, so a site can stay open in part and closed in part.
• It is a genuine first: four major labs agreeing on one mechanism, in public, on the same day.

Cons:

• It is voluntary, unenforced, and not yet a finished standard — the draft expires in May 2027 unless updated.
• Timelines are staggered, so protection is partial until 2027, and Meta has not committed at all.
• It does nothing about content already collected or already trained into models.
• It covers training only, not grounding or search-AI features, which need separate controls.
• It can be misunderstood as a crawl block, and a misplaced group in robots.txt can cause real damage if it is confused with a search rule.

FAQs

What is the new AI training opt-out in robots.txt?

It is a shared policy token, __usage_ai-training, backed on October 30, 2026 by OpenAI, Google, Microsoft, and Anthropic. Sites add it to robots.txt to state that their content should not be used for generative AI training.

Does the AI training opt-out block Google Search or crawling?

No. It is a training preference, not a crawl or indexing rule. Google says it does not affect inclusion in Google Search, and Microsoft says it does not change eligibility for ordinary Bing results.

When will companies start honoring the new signal?

Anthropic says it has already started. OpenAI aims to implement it in the coming months. Microsoft starts on February 1, 2027, and Google aims to support it in 2027, in early 2027 per its crawler docs.

Do I still need GPTBot and Google-Extended rules?

Yes, for now. The old company-specific controls keep working alongside the new signal and are the only ones with live effect until each company switches the shared token on. Keep them until 2027.

Does opting out remove content already used for AI training?

No. The signal looks forward only. Updating robots.txt does not remove material already collected into datasets or already trained into model weights.

Does the opt-out stop my pages appearing in AI answers?

Not by itself. Training and grounding are separate choices. Microsoft offers a separate grounding preference, and Google has a separate Search generative AI control in Search Console for AI Overviews and AI Mode.

Verdict

Add the shared token if you want a training opt-out — but add it with your eyes open. The two-line group costs almost nothing, it future-proofs your site for 2027, and it is the first time the major labs have agreed to read one preference the same way. Keep your existing per-company rules until each company’s start date, set grounding and search-AI controls separately if those matter to you, and re-check the draft before it expires in May 2027.

What this announcement really gives publishers is simpler plumbing, not a settlement of the training-data fight. The fight itself — over consent, compensation, and what “publicly available” should mean for AI — will be decided in courts, in regulation such as the EU AI Act’s Code of Practice, and in the commercial deals publishers strike. A robots.txt token is how a small site records its side of that argument in a form machines can read. That is worth doing. It is not worth mistaking for the argument being over.

📚 Read next: Claude Duet: Anthropic’s proactive AI agent explained · Sweden blocks ChatGPT and Claude on government PCs · AI statistics 2026: 60 key numbers

Last updated: October 31, 2026. Methodology: This article was researched on October 31, 2026 from the October 30, 2026 announcements by OpenAI, Google Search Central, and Microsoft Bing, the IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, and reporting by TSN, Search Engine Land, and Search Engine Roundtable, cross-checked against one another where timelines differed. No crawler behavior was hands-on tested; implementation dates are the companies’ stated plans, not verified live behavior. This article is general information, not legal advice. It is not sponsored, and no company mentioned paid for placement. Learn more on our About page, or contact us with a correction. More from the blog: The AI Tools homepage and the AI News archive.