Cloudflare launches way to charge AI bots for crawling sites

Cloudflare announced a new tool on Tuesday that can allow publishers to charge AI bots for scraping websites.

That could prove to be a major change to how the internet has functioned so far in the AI age — models have, by default, scraped the internet with abandon and without permission, often to the chagrin of content owners. Cloudflare is a major infrastructure provider, meaning large swaths of the internet will get access to this tool.

The “pay per crawl” feature was part of Cloudflare’s announcement on Tuesday that the company is now the first internet infrastructure provider to “block AI crawlers accessing content without permission or compensation, by default.” The tool, which is currently in beta, allows site owners to charge a fee each time an AI bot wants to “crawl” its website for information.

The idea behind this push from Cloudflare is to put power back in the hands of the people who make the content that AI uses to train. (Full disclosure: Ziff Davis, which owns Mashable, was among the many publishers quoted in Cloudflare’s press release that supported a permission-based approach to AI bots.)

Wrote Cloudflare in its press release:

“For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information. That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content.”

We’ll see how it evolves, but the tool could prove to be useful for people who create original content on the internet.