Cloudflare's new AI crawler categories can accidentally block Googlebot
Cloudflare rolled out a three-tier crawler management system — Search, Agent, and Training — with defaults activating Sept. 15. Sites that opt to block AI training can inadvertently block Googlebot, because Google's crawler is used for both search indexing and Gemini training and doesn't cleanly separate the two. The controls apply to all Cloudflare sites, not just Enterprise.
Before Sept. 15, pull your Cloudflare bot rules and confirm Googlebot is explicitly allowed under Search — the training toggle is where sites will accidentally deindex themselves. For regulated clients, document the crawler policy alongside your robots.txt as part of change control.