Cloudflare, which sits in front of roughly a fifth of the world's websites, has set the AI industry a hard deadline. From September 15, 2026, its default settings will block "mixed-use" crawlers — bots that blend search indexing, AI agent activity, and model training — from any page that carries ads. The defaults apply to new customers, new sites from existing customers, and the entire free tier.
The unnamed but unmistakable target is Google, whose flagship Googlebot crawls for Search and AI features alike. Cloudflare argues this gives the search giant roughly twice the data access of rival AI firms, because publishers cannot stay discoverable without also feeding Google's AI. The company is also relaunching Pay Per Crawl as "Pay Per Use": publishers get paid when their content shapes an AI answer, not just when it is fetched, with Ceramic.ai and You.com as first partners.
The escalation is already spreading. USA Today's parent network says it is prepared to delist from Google within six to twelve months absent a licensing deal, and newsletter platform Beehiiv now lets its creators block Googlebot outright. Cloudflare's own data adds an efficiency argument: over half of AI crawl traffic goes to re-fetching pages that have not changed.
