Web infrastructure provider Cloudflare has been pushing for increased regulation in the AI space since earlier this year when it launched a marketplace that allows websites to charge AI bots for scraping their content.
The company’s chief executive, Matthew Prince, is in London to speak to Britain’s Competition and Markets Authority (CMA) and said he is proposing stricter rules for how Google can compete in the AI race given its search dominance.
Earlier this month, the CMA granted Google special status in the search and advertising markets due to its “substantive and established” position. The move will allow regulators to impose stricter regulations not only on search and advertising, but also on areas such as Google’s AI Overview, AI Mode, Discover feed, Top Stories, and News tab.
Prince said that while Cloudflare is not in the AI business itself, it has numerous relationships with AI companies, which puts it in a good position to make recommendations.
“We’re not directly involved in the fight. We’re not an AI company,” Prince said in a speech at the Bloomberg Tech Conference in London this week. “We are not a media publisher, but a network between media publishers. 80% of AI companies are our customers,” he added.
Cloudflare’s president believes Google should compete on a level playing field with other AI companies, but he said that’s not the case right now. Rather, Google uses its existing web crawler, in addition to its search engine, to crawl content for AI products and services. This gives Google an unfair advantage, Prince said. (Prince is referring to Googlebot, which crawls for search, including AI features such as AI Overview.) Google spokesperson Ned Adrians said that site owners can use Google Extended to opt out of content used to train AI products, but this does not affect a site’s inclusion in Google Search. We think that’s about right, as it’s likely that you want to opt out of the feature completely.
“Google is saying, ‘We have an absolute God-given right to all the content in the world, even if we don’t pay for it, because look what we’ve done over the last 27 years,'” Prince explained. “And they say you can use the same crawlers that you use for search to power your AI systems. And if you want to opt out of either, you have to opt out of both,” he pointed out.
tech crunch event
san francisco
|
October 27-29, 2025
This is clearly not feasible for most companies, especially those in the media business where losing search means losing about 20% of their revenue, the executive said.
“But it gets worse. Blocking Google’s crawlers also blocks Google’s ad safety team, which means ads will stop working on all platforms. This is not just the beginning,” Prince added.
Google bundles its own crawler, which gives you access to content that you would have to pay for from other companies like Anthropic, OpenAI, and Perplexity.
“The problem is we’re effectively handing the game over to Google,” he said.
The solution, Prince said, is to foster more competition in the market, potentially with thousands of AI companies competing to buy content from thousands of media companies and millions of small businesses. He suggested that the UK’s CMA flagging Google as a potential regulatory target was a thoughtful move and shows they recognize Google’s unique advantages.
Cloudflare also provided CMA with data showing how Google’s crawler works and why it’s nearly impossible for other players to replicate Google’s success.
Prince isn’t the only one sharing this opinion these days. Last month, Neil Vogel, CEO of People, the nation’s largest digital and print publisher that operates more than 40 media brands, said essentially the same thing. In an interview, he called Google the “bad guy” and said the way its crawlers are combined has forced media companies to rely on Google to crawl their sites looking for AI content.
Vogel’s company has adopted Cloudflare’s solution to block AI crawlers that don’t pay, and he said contract negotiations are underway with several large LLM providers and insisted the system is working.
Updated to add Google’s comment.
