Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Key Takeaways:

• Perplexity recrawls websites with stealth, undeclared crawlers when initially blocked from crawling (impersonating Google Chrome on macOS)

• Findings based on test of Perplexity crawling behavior

• Cloudflare detected 3-6 million undeclared crawler hits daily compared to 20-25 million hits from the official crawler

Source: https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/