A major conflict between Perplexity, an AI-powered search startup, and Cloudflare, a web infrastructure firm, is shaping internet governance in 2025. At the center is whether AI agents fetching real-time web content on user request should be treated as bots or as human proxies. Perplexity claims that its agents will not act unless initiated by the users, and Cloudflare suspects it of going around anti-scraping policies. This face-off points to more tension that surrounds the role of AI in restoring web access and monetization. The debate suggests that blocking AI bots on 45 percent of websites can lead to an increasingly rule-based web that breaks into more fragmented pieces unless standards transform.
Perplexity and Cloudflare Clash Over What Defines a Bot
Perplexity uses AI agents to retrieve online content in real time, responding to specific user queries. These agents do not continuously crawl the web or train on the data; they simply fetch and summarize content. Perplexity argues this is user-directed behavior, functionally equivalent to a human browsing a page, and thus should not be classified as automated scraping.
Cloudflare, however, claims Perplexity is stealth crawling websites, violating robots.txt directives, and evading detection by disguising its identity through user-agent switching and IP rotation. The security firm states it has recorded millions of requests attributed to Perplexity daily, even on domains explicitly blocking AI agents.
Perplexity refutes this, attributing the volume to BrowserBase, a third-party tool it occasionally uses. It states that fewer than 45,000 daily requests originate directly from its systems and claims Cloudflare’s analysis conflated BrowserBase’s broader traffic with Perplexity-specific usage.
Such controversy addresses a key question: when an AI system uses the internet on behalf of the user, is it a crawler or a digital assistant? This technical neutrality in the state of the current bot-detection infrastructure turns this conflict into a proxy war over the bigger issues of how to treat various forms of AI on the Internet, particularly when the technology is acting under human direction.
Governance, Economics, and the Fight Over Web Access Rules
The dispute reveals deep problems in how the internet governs AI traffic. Web infrastructure is still built around detecting crawlers and bots, not intelligent agents acting as user extensions. This outdated model leads to misclassifications, as in Cloudflare’s case, and blocks user-driven AI systems like Perplexity from functioning freely. With 45% of websites now restricting AI bots, according to multiple industry reports, the risk of a fragmented, technical, and economic internet is real. Publishers rely on ad revenue from human visitors. AI agents don’t click ads, view banners, or interact with paywalls in the same way. Some argue this could justify access fees for AI-driven interactions. One proposed solution is a new standard for AI agents in robots.txt, allowing sites to distinguish between general bots and human-initiated AI requests. Others suggest micropayments per AI query, potentially via blockchain or stablecoins, to offset lost revenue.
The debate also extends to ethics and legality. If an AI agent accesses a site that has explicitly disallowed bots, who’s liable, the provider or the end user? The evolving role of AI as both a tool and an actor forces a reevaluation of consent, fair use, and access rights in the internet’s next era.
Future of AI Agent Access and Online Rules
The Perplexity-Cloudflare conflict signals a turning point in how AI and web infrastructure coexist. AI agents acting on user requests challenge the old rules that governed passive bots and crawlers. Without clear standards, misidentification, gatekeeping, and legal ambiguity will intensify. Solving this is potentially in updated protocols, potentially extending the format of robots.txt to be aware of AI, or a new economic model where the costs of making something available to a publisher would be acceptable, where there is AI on it.
About The Author
