Major AI Platforms Implement Shared Protocol to Block Training Scraping
What happened
On October 30, 2026, Google, OpenAI, Microsoft, and Anthropic announced joint support for a new entry in the standard robots.txt file. This protocol, called __usage_ai-training, allows publishers to block AI training across all four platforms simultaneously. Previously, publishers were required to name each crawler individually to restrict content scraping.
From techtimes.com
Why it matters
The new standard provides a unified mechanism for publishers to control how their web content is utilized by generative AI models. This shift directly impacts the availability of web content used for training these large language models.
From techtimes.com
Who's involved
- AnthropicOne of the four dominant Western AI platform operators implementing the shared protocol.
- GoogleOne of the four dominant Western AI platform operators implementing the shared protocol.
- MicrosoftOne of the four dominant Western AI platform operators implementing the shared protocol.
- OpenAIOne of the four dominant Western AI platform operators implementing the shared protocol.
Who could feel it
Possible knock-on effectsThese are possibilities Brind reasoned out, not predictions, and not advice. Most are not stated in any report.
- AnthropicSpeculative
Anthropic might face increased costs for data acquisition if training content becomes restricted.
- GoogleSpeculative
Google might face increased costs for data acquisition if training content becomes restricted.
- MicrosoftSpeculative
Microsoft might face increased costs for data acquisition if training content becomes restricted.
- OpenAISpeculative
OpenAI might face increased costs for data acquisition if training content becomes restricted.
Possible ripples
Who this could reach next, traced step by step from the reporting. Possibilities worth considering, not forecasts.- Moonshot AI may face increased data acquisition costs from new AI protocolsMoonshot AI may face increased data acquisition costs from new AI protocolsPossible
- New AI training protocol may help New York Times fight unauthorized scrapingThe new AI training protocol may help New York Times fight unauthorized scrapingPossible
- Salesforce's AI integration may be affected by new data opt-out rulesSalesforce's AI features may face changes if training data sources are restricted.Possible
- Meta may limit data available for AI training via new protocolMeta may limit data available for AI training via new protocolPossible
Keep exploring
The entities involved
-
Anthropic
American artificial intelligence corporation
-
Google
American multinational technology company, a subsidiary of Alphabet Inc.
-
Microsoft
American multinational technology corporation
- The Dutch government is restricting data center sizes while Equinix plans a large data center in Amsterdam-Zuid, where Microsoft is conducting a project demonstration.
- Sam Altman's background and OpenAI's funding structure are detailed, including Microsoft's $40B investment and early investments by Altman in Airbnb.
-
OpenAI
American artificial intelligence research organization