Cloudflare announced on July 1, 2026 new rules for blocking AI crawlers: from September 15, AI companies must separate their web indexing bots from those used for model training and agent serving. Crawlers combining both functions — so-called mixed-use crawlers — will be blocked by default on all ad-supported pages served by Cloudflare, unless the site owner explicitly changes that setting.
Key takeaways
- From September 15, 2026, mixed-use crawlers blocked by default on ad-supported sites in Cloudflare's network
- The change applies to new customers, new sites created by existing customers, and all existing free-plan customers
- Cloudflare expands Pay Per Crawl into Pay Per Use — billing for actual content usage, not just fetching
- Cloudflare data: over 50% of AI crawler traffic comes from re-fetching pages that have not changed
- First partners: Ceramic.ai and You.com pay publishers for content appearing in their AI service results
The problem with mixed-use crawlers
Search indexing crawlers and AI training or agent-serving crawlers serve different purposes, but many AI companies send a single bot that does everything at once. Publishers who want to remain visible in search results must allow indexing — and with a mixed-use crawler, their content also ends up in model training without any compensation.
Cloudflare points out that the world's largest search engine (Google) has access to roughly twice as much content as other AI companies, because publishers wanting search visibility cannot block Googlebot without harming their SEO. Google has pushed back on this criticism by citing Google Extended — a separate bot that lets site owners opt out without falling out of the search index. However, Googlebot still crawls pages for AI Overviews and AI Mode without a separate opt-out.
The scale of the issue is illustrated by Cloudflare's own data: over 50% of AI crawler traffic consists of re-fetching pages that have not changed at all — wasting publisher bandwidth and compute resources.
Pay Per Crawl evolves into Pay Per Use
Cloudflare launched Pay Per Crawl a year ago as a marketplace where site owners can set prices for AI bot scraping. The model is now evolving into Pay Per Use: billing occurs not when a page is fetched, but when that content actually generates value — when it appears in an AI agent's response or a search result.
The first companies participating in the program are Ceramic.ai and You.com. When a publisher opts in, they receive payment each time their content appears in Ceramic.ai's AI search results or when You.com accesses a piece of their premium content. Other AI companies can customize the billing model to fit their own services.
Scope of the default setting changes
The new default settings will apply from September 15 to all new Cloudflare customers, new sites created by existing customers, and all existing customers on free plans. Current paid-plan customers can keep their existing settings. Site owners who want to allow mixed-use crawlers must explicitly enable this in the Cloudflare dashboard.
Why this matters
Cloudflare handles traffic for a significant portion of the global web — the company serves over 20% of global HTTP traffic. Its decision to block mixed-use crawlers by default is a practical enforcement of the separation between search and AI bots that neither regulation nor technical mechanisms had previously enforced.
For publishers, this signals that the ability to charge AI companies for content is becoming infrastructure, not just an aspiration. Pay Per Use may set a precedent for similar solutions on other CDN and hosting platforms.
For AI companies, the change requires technically separating crawler infrastructure and is effectively an initial pricing of open-web access as training data — a resource previously treated as free. The open question is enforcement: AI companies may attempt to circumvent the block by rotating bot identifiers or using intermediaries.
What's next
- The new default settings take effect on September 15, 2026.
- Matthew Prince, Cloudflare's CEO, indicated that AI companies that have not separated their crawlers by that date will be interpreted as acting in bad faith toward the publisher ecosystem.
- The Pay Per Use program is open to additional AI companies — Cloudflare has not disclosed how many firms are on the waiting list.
