Cloudflare says Perplexity’s crawlers hid behind changing identities

# Cloudflare says Perplexity’s crawlers hid behind changing identities

The company says the answer engine ignored no-crawl signals and shifted user agents, IPs and ASNs to keep crawling.

Cloudflare is accusing Perplexity of more than ordinary scraping. In a detailed blog post, the company says it observed stealth crawling behavior in which Perplexity initially used a declared user agent, then changed user agents, IPs and autonomous system numbers when blocked. Cloudflare says the pattern appeared to be an attempt to obscure the crawler’s identity after a site expressed a no-crawl preference.

That is a serious claim because it goes beyond the usual arguments about volume or rate limits. The issue here is transparency. Cloudflare says websites had stated their preferences clearly, but Perplexity’s crawlers continued anyway and appeared to alter their identity to evade detection. The company says the crawler also ignored - or in some cases did not even fetch - robots.txt files. If accurate, that suggests a deliberate effort to work around site-owner instructions rather than a mistake in configuration.

The blog post is also interesting because it contrasts behavior. Cloudflare says it observed ChatGPT’s declared crawler fetching robots.txt and stopping when disallowed, and again stopping after encountering a block page. That comparison is part of the argument the company is making about what responsible crawling should look like: clear purpose, visible identity, respect for site preferences and no attempts to slip past network-level blocks.

Cloudflare says it responded by de-listing Perplexity as a verified bot and adding heuristics to its managed rule set. That matters because bot management is not just about stopping nuisance traffic; it is also about defining acceptable behavior on the web. Once a platform provider says a crawler is operating stealthily, the situation becomes a trust issue as much as a technical one.

Perplexity’s role in the market also helps explain why the issue drew attention. As an AI answer engine, it depends on large-scale access to web content, but access is not the same as permission. Cloudflare’s post argues that crawlers should serve a clear purpose, be transparent and respect robots.txt and other signals. That standard is increasingly relevant as more AI products rely on web retrieval and automated summarization.

The broader implication reaches beyond Perplexity. If a popular AI service can be accused of hiding its identity after a block, site owners will push harder for stronger defenses and clearer bot policies. That could make crawling more adversarial at the same time that AI systems are becoming more dependent on the open web. In other words, the tension here is not just about one company; it is about whether the web can keep functioning on the assumption that bots will play by visible rules.

For now, the evidence supports a stark summary: Cloudflare says Perplexity’s crawlers did not respect site preferences and tried to keep going by changing how they appeared to the sites they visited. That puts the company in the middle of a broader debate over what AI crawlers owe the web in return for access to it.