Cloudflare has announced the launch of a new marketplace called Pay per Crawl, currently in private beta, that enables websites to charge bots for data scraping. This innovative platform allows content creators to set prices for access to their data, while companies operating web crawlers can choose how much they're willing to pay.
Serving as a trusted intermediary, Cloudflare handles the transactions between publishers and bots, bringing a new layer of transparency and control to automated data access. Site owners will also have the flexibility to allow bot access for free or to block it entirely, depending on their preferences.
This initiative reflects a broader shift toward the monetization of digital content and the need for a more regulated, consensual ecosystem for data scraping—especially in the age of AI and large-scale data collection.
The news was first reported by TechCrunch.
