Cloudflare Now Sorts AI Bots Into Three Kinds, and Googlebot Can Get Caught
Cloudflare’s earlier Block AI bots option treated AI bots as one group. On July 1 Cloudflare added three separate settings, and what you pick now affects more than it seems.
Search, Agent, Training
Search bots index content so they can answer questions about it later. Agent bots act in real time on behalf of a person, such as a chat assistant that loads a page or a browser agent. Training bots collect content to train or fine-tune models.
For each category you choose one of three options, which are to block on all pages, to block only on pages that show ads, or not to block at all. The options are available on every plan, including the free one. According to Cloudflare’s announcement, from September 15 new domains get Training and Agent blocked on pages with ads by default, and Search allowed. Some third-party write-ups say the same defaults apply to existing free accounts that haven’t changed their settings, although the announcement itself only mentions new domains.
Googlebot gets caught
The hardest case is crawlers that do several things at once. Googlebot, Applebot and Bingbot both index sites for search and collect data for training. Cloudflare says that from September 15 such crawlers are blocked by customers who block Training, even if Search is allowed. Cloudflare sees this as a reason for companies to split their crawlers by purpose, and its July report singles out Google, which uses one crawler for everything. Owners could opt out before September 15 by marking in the Security settings that they want no change for Training crawlers that also crawl for Search. If your site lives on advertising or organic traffic, open the zone settings and look at what is set now.
Why robots.txt is not protection
Robots.txt is an agreement from 1994 that nobody is obliged to follow. In August 2025 Cloudflare described how Perplexity, when its declared crawlers were blocked, kept visiting sites with a crawler that presented itself as Chrome on macOS. Cloudflare put it at 3 to 6 million such requests a day, next to 20 to 25 million from the declared crawler. It removed Perplexity from its list of verified bots and added rules to block the undeclared crawling. Cloudflare sells this kind of protection, so its conclusions are worth reading with that in mind.
How much of this traffic there is
According to Cloudflare’s data for June 2026, 52% of crawler requests are for model training, up from 22% in spring 2025. More than a third comes from mixed-purpose bots, pure search crawling is a small and shrinking share, and more than half of all internet traffic is now non-human.
For a services site without ads the choice is probably simple, so keep Search and decide on the rest as you like. Publishers that run on ads have more to lose, and a mistake can cost them their place in Google. We would start with the server logs, because before blocking anything it helps to see who actually visits the site.
