Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:
- Allow: Grant the crawler free access to content.
- Charge: Require payment at the configured, domain-wide price.
- Block: Deny access entirely, with no option to pay.
Although this is old news, I still found it interesting. Also, I like the “AI Labyrinth” feature of Cloudflare to block AI bots.
Well, cockmushroom, then you’re building a pretty expensive dashboard. I still don’t see how they are going to differentiate between normal browsing and bot crawling. I don’t think that’s an exact science.