Cloudflare has unveiled a new AI scraping policy that will enable website owners themselves to ‘decide which crawlers to allow.’ Photo Credit: Omar Lopez-Rincon
Bad news for artificial intelligence giants: Cloudflare has officially updated its AI scraping defaults in an effort to halt the “use of original content without permission.”
The self-described “leading connectivity cloud company” revealed the move – and the resulting “new business model” that it’s anticipating – in a formal release. With the change in place, “website owners can choose if they want AI crawlers to access their content,” Cloudflare summed up.
“AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow,” Cloudflare continued.
When it comes to unauthorized (and non-compensated) gen AI training, it’s hardly a secret that authors, music rightsholders, platforms like Reddit, and plenty of others are far from thrilled with the apparently widespread practice.
But Cloudflare also emphasized the traffic-related fallout associated with AI-powered search results. “AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content,” the entity indicated.
Stated differently, the AI scraping change isn’t directly tied to the music world; Cloudflare’s formal release explores expected benefits for newspaper and magazine publishers in particular. (There’s some ownership overlap here: Advance Publications owns Condé Nast as well as 30% of the aforementioned Reddit, and the CEOs of the latter two companies provided statements for Cloudflare’s release.)
Nevertheless, the policy pivot definitely ties into the music industry, where alleged training free-for-alls and purportedly infringing outputs remain a key focus. It’s been about one year since the majors publicly warned AI developers to steer clear of no-license training.
Consequently, Universal Music-partnered ProRata AI and the major label itself are among the many backers of Cloudflare’s approach.
“We welcome this new initiative from Cloudflare,” weighed in Universal Music COO Boyd Muir, “that will help address the indiscriminate, disruptive, and unauthorized scraping of both creative and commercial IP by AI model developers and support new licensing.
“At UMG, we have always embraced innovation and new technologies, and firmly believe that AI, when used ethically, transparently, and respectfully of copyright and human creativity, has the opportunity to introduce significant new avenues for creativity and future monetization,” the exec concluded.
On the monetization and licensing front, Suno and Udio are reportedly in talks to settle the infringement suits levied against them by the majors. Plus, a mediation deadline extension in music publishers’ Anthropic complaint was made official last month.
Closing with an overview of the policy-shift specifics, Cloudflare spelled out that “AI companies will now be required to obtain explicit permission from a website before scraping.” Meanwhile, each new domain will at the time of sign-up “be asked if they want to allow AI crawlers.”
Furthermore, since Cloudflare started allowing clients “to block AI crawlers in a single click” in September 2024, “[m]ore than one million customers” have taken advantage of the option, the company noted.
