Cloudflare has officially updated its default settings to block all AI bots across its CDN services. The decision marks a significant shift in how the company approaches AI-related web scraping and content protection. According to the official announcement, this move is designed to prevent unauthorized use of web content for training large language models.
Website owners using Cloudflare's services will still have the option to manually allow access to AI bots if they choose. Additionally, Cloudflare is introducing a new “pay-per-crawl” feature aimed at content creators. This feature enables selected users to set pricing terms, allowing AI companies to pay for access to their site data—offering a more controlled and transparent alternative to unrestricted scraping.
Cloudflare’s CEO, Matthew Prince, emphasized the importance of preserving online originality:
“Original content is what makes the internet one of the greatest inventions of the last century, and we need to stand together to protect it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators.”
Cloudflare has been vocal on this topic since 2023, when it first implemented AI bot blocking via the robots.txt file. In 2024, it introduced enhanced tools to allow users to block even those bots that ignored robots.txt. Now, the company is making AI bot blocking the default setting for all new users—whether the bots are accessing content for training, inference, or research purposes.
This update signals a growing trend in the tech industry: the push to give creators more control over how their digital work is accessed and used by artificial intelligence tools.
