• Cloudflare's pay-per-crawl feature: It is in private beta with a small number of publishers like AdWeek, etc. Participating publishers can set their own prices for AI crawlers to scrape content. CEO Matthew Prince said it ensures the Internet's survival in the age of AI by putting power back in creators' hands while helping AI companies innovate. Some publishers are optimistic it can stop copyright infringement. Any content creator can sign up and potentially be compensated. Publishers in the beta can choose which bots access their sites and charge some bots while letting others scrape for free.
• AI companies' role: For the plan to work, AI companies must sign up. Cloudflare has partnered with them, and they can benefit from long-term collaboration with creators and avoid wasting money on poor data sources. But it depends on AI companies agreeing to pay publisher prices and Cloudflare detecting the bots.
• Future of the system: The pay-per-crawl system is expected to evolve. Publishers may charge different rates for different paths or content types and introduce dynamic pricing. AI companies may be incentivized to create agents to crawl the web and seek the best content deals.
• Blocking AI crawlers by default: After rolling out a feature to allow website owners to block AI crawlers with one click last September, Cloudflare now sets new customers' domains to block all known AI crawlers by default. This is a transition from opt-out to permission-based models to give users more control over their content and prevent blocking of search engine bots. Cloudflare's blog emphasized the importance of original content and the risks of AI scraping. It should be noted that Condé Nast, which owns Ars Technica, is a partner in the pay-per-crawl beta.