Cloudflare unveils new AI content controls, analytics and monetisation tools for website owners

Cloudflare has introduced new AI crawler classifications, analytics capabilities and commercial partnerships to help website owners manage AI access, improve content discoverability, optimise crawling efficiency and explore new monetisation models in the emerging agentic internet.

Cloudflare has announced a series of new tools, analytics capabilities and commercial partnerships aimed at giving website owners greater control over how artificial intelligence companies access, use and monetise online content. The company said the initiatives are designed to support what it describes as the emerging "agentic Internet," where AI agents increasingly interact with websites for search, commerce and content consumption.

According to Cloudflare, more than half of all web requests are now generated by automated agents and bots, reflecting a significant shift in internet traffic patterns. As AI becomes a primary interface for discovering information, the company said website owners need more transparency and flexibility in determining whether AI systems can crawl their content for search, agent use or model training.

Cloudflare's latest offering includes new crawler classifications, enhanced analytics for monitoring AI traffic, faster AI search capabilities and commercial frameworks intended to enable content creators to receive compensation when their work contributes to AI-generated responses. The company said these measures are aimed at balancing content discoverability with intellectual property protection while improving efficiency for both publishers and AI companies.

"Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem,” said Matthew Prince, co-founder and CEO of Cloudflare. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”

New AI crawler policies and monetisation initiatives

Cloudflare said it plans to introduce new default settings and crawler classifications from September 15, 2026, following a two-month consultation period with publishers and AI companies. Under the proposed changes, new websites using Cloudflare's services will, by default, allow AI-powered search while blocking AI model training and agent-based usage on pages containing advertisements. Mixed-use crawlers that do not distinguish between search, training and agent functions will also be blocked from such pages unless customers choose different settings.

The company is also expanding its analytics capabilities through a new Attribution Business Insights dashboard, enabling publishers and businesses to measure how AI bots consume their content and understand how much referral traffic AI platforms generate. Cloudflare said the platform is also intended to support Answer Engine Optimization (AEO), helping organisations assess how frequently and prominently their content is cited by AI-generated responses.

In addition, Cloudflare announced plans to enhance AI crawling efficiency by signalling when web pages have changed, reducing unnecessary repeat crawls. The company estimates that more than half of AI crawler traffic is currently spent revisiting unchanged pages, leading to increased bandwidth and computing costs.

Cloudflare is also extending its Pay Per Crawl initiative into a new Pay Per Use model, under which publishers would be compensated when their content is used to generate value rather than simply being crawled. The company said it is working with partners including Ceramic.ai and You.com to support different commercial models that allow AI platforms to compensate content owners based on content usage.

Alongside the product announcements, Cloudflare highlighted progress made over the past year through initiatives such as AI Crawl Control, Web Bot Auth and collaborations with publishing platforms and payment providers. The company said these developments are intended to provide the technological foundation for a more transparent and sustainable AI ecosystem, while several publishers and AI companies, including beehiiv, Ceramic.ai, Condé Nast and Patreon, have expressed support for its approach to improving content control, discoverability and creator compensation.