OpenAI, Google, Microsoft, and Anthropic agree on one way for sites to say no to AI scraping

Publishers trying to keep their content out of AI training data have had to fight that battle one company at a time. That's about to change. OpenAI, Google, Microsoft, and Anthropic have all said they will support a shared technical standard that lets website owners opt out of AI scraping with a single signal, rather than configuring separate rules for each crawler, as Search Engine Roundtable first reported.

The mechanism is a "policy token," proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force, the body that develops standards for how the internet actually works. The token would sit inside a site's robots.txt file, the plain-text file that has quietly governed web crawling for decades. Robots.txt already tells search engines and archivers what they can and can't touch. This would extend that same file to cover AI training, so a publisher could set one preference that every participating AI company's crawler is expected to honor.

Right now, that's not how it works. If a publisher wants to block OpenAI's crawler but allow Google's, or block all of them from training while still allowing search indexing, they generally have to communicate those preferences separately to each company, through whatever tools that company happens to offer. A shared token would turn that patchwork into a single switch.

Why this is happening now

AI crawlers have become a persistent headache for the people who make the content those models are trained on. Every major AI company runs bots that sweep the web, and publishers have increasingly tried to push back against being scraped without compensation or consent. A few approaches have already emerged. The Really Simple Licensing standard lets publishers charge crawlers for access to their sites. Cloudflare, which sits in front of a huge share of the web, started blocking AI crawlers by default earlier.

Those efforts chip away at the problem but don't solve the coordination issue: every publisher still has to deal with every AI company individually, and every AI company has to interpret whatever signals each publisher happens to be using. A shared, standardized opt-out addresses that directly, at least for the companies that agree to respect it.

What the companies are actually committing to

OpenAI, Google, Microsoft, and Anthropic have all said their crawlers will stop accessing a site if they detect the new token. That's the baseline commitment, and it's notable mainly because it's four of the largest AI developers agreeing to the same mechanism rather than building their own competing versions.

Microsoft is going a step further. It plans to offer more granular controls that distinguish between two different uses of scraped content: AI training, where content feeds into building a model, and AI grounding, where a model pulls from a verified source to check its answers and reduce hallucinations — the made-up or inaccurate outputs that are one of AI's persistent weaknesses. That distinction matters to publishers who might be fine with their content grounding an AI's answers (which can drive traffic back to them) but not fine with it being absorbed into a model's training data permanently. Microsoft also notes that a site's AI preferences won't affect whether it shows up in Bing's search results — training and grounding opt-outs are separate from search indexing.

The timeline isn't immediate

This isn't flipping on tomorrow. Anthropic has already adopted the signal, which tracks with Kelley's role in proposing it. Google and Microsoft say they'll support it next year. OpenAI is vaguer, saying only that it will implement the standard in the "coming months."

That staggered rollout means publishers won't get the full benefit of a universal opt-out right away. Setting the token today would stop Anthropic's crawlers but not necessarily the others until their respective timelines catch up. And because this is a voluntary standard rather than a legal requirement, it only works insofar as companies choose to honor it, something publishers have already learned the hard way with robots.txt itself, which plenty of crawlers from smaller or less scrupulous operators simply ignore.

Still, getting four of the industry's biggest players to converge on one mechanism, through a formal standards body rather than four separate opt-out tools, is a meaningfully different posture than what publishers have been dealing with. It won't end the broader fight over compensation, consent, and how AI companies use web content. But it does promise to make the basic question — can your crawler touch my site — something publishers can answer once instead of repeatedly.

What happens next largely depends on whether other AI companies beyond this initial group adopt the same token, and whether the IETF process produces a stable, permanent specification rather than something these companies could quietly deprioritize once the standards process runs its course.