Cloudflare Introduces New AI Crawling Rules That Require Payment for Publisher Content

Cloudflare has introduced a significant policy change to reshape how AI companies access online content. The company announced on Wednesday that, beginning September 15, 2026, AI firms will be expected to separate web crawlers used for traditional search indexing from those designed for AI training and agent-related tasks.

Under the new default settings, Cloudflare will automatically block “mixed-use” crawlers from accessing webpages that display advertising. These are crawlers that combine multiple purposes—including search indexing, AI agent operations, and model training—within a single bot. Website owners will still be able to modify these settings if they choose to allow access.

The updated defaults will automatically apply to all new Cloudflare customers, newly created websites under existing customer accounts, and all websites using Cloudflare’s free tier, according to the company.

The policy could significantly affect how AI developers gather online content for training large language models and supporting AI-powered services.

Cloudflare says most publishers still want their websites to remain visible in traditional search engines and, in many cases, accessible to AI services. However, many content creators also want stronger protections to prevent their intellectual property from being freely collected and reused without compensation.

The company specifically pointed to what it described as the “world’s largest search engine”—an apparent reference to Google—claiming that it enjoys access to roughly twice as much web content as many competing AI providers because website operators often struggle to remain discoverable in search while avoiding AI-related data collection.

Google has previously challenged that characterisation, arguing that it already provides publishers with a dedicated crawler called Google Extended. That tool allows website owners to opt out of having their content used for AI model training and AI products such as Gemini Apps and Vertex AI without affecting their visibility in Google Search. However, Google’s primary crawler, Googlebot, continues to power Search features, including AI-generated experiences such as AI Overviews and AI Mode.

Cloudflare co-founder and CEO Matthew Prince said the company believes further action is necessary as internet traffic increasingly shifts away from humans.

“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Prince said, referencing Cloudflare’s recent finding that automated bot traffic has surpassed human-generated traffic online sooner than previously expected.

Prince added that the company’s latest products and industry partnerships are intended to provide publishers with greater transparency, stronger commercial opportunities, and improved control over how AI companies access their content.

“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate search from agent use and training,” he said.

While Cloudflare continues to expand its AI offerings, it has also introduced several products designed to help publishers maintain greater control over their digital assets. Among those initiatives is Pay Per Crawl, a marketplace introduced to allow website owners to charge AI companies for scraping their content.

The company also announced that Pay Per Crawl is evolving into a broader system called Pay Per Use. Under the updated model, publishers will be compensated when their content actively generates value through AI applications rather than simply when it is crawled.

Cloudflare believes the change could also reduce unnecessary bandwidth and computing costs for publishers. Internal data indicate that more than half of AI crawler traffic currently consists of repeated fetches of unchanged webpages.

To launch the new payment model, Cloudflare is partnering initially with Ceramic.ai and You.com. Publishers who participate can receive compensation whenever their content appears within Ceramic’s AI-powered search results or when You.com accesses premium material through participating websites.

According to Cloudflare, additional AI companies will be able to customise the PaaS framework to align with their products and business models as adoption expands.