OpenAI, Google, Microsoft, and Anthropic are backing a new standard that lets websites refuse to have their content used for AI training. The mechanism works through a token called __usage_ai-training in a site's robots.txt file, paired with the rule "Disallow: /", allowing publishers to block AI scrapers while still permitting search engines to index their content normally.
Anthropic has already begun implementing the feature. Google and Microsoft plan to offer support by 2027, while OpenAI expects to adopt it within the coming months. The standard addresses growing concerns from publishers about unauthorized use of their content in large language model training.