Cloudflare Now Sorts AI Crawlers Into Three Buckets. You Have to Pick a Policy for Each.

Cloudflare just changed a default that most teams never think about, and it is going to quietly reshape who gets to read your website.

Starting September 15, 2026, Cloudflare stops treating "AI bots" as one blob. It sorts automated AI traffic into three categories: Search, Agent, and Training. Each gets its own policy. And for new sites and all free-tier customers, the default on pages that show ads flips to blocking Training and Agent crawlers while leaving Search allowed.

If you run anything on Cloudflare, and a huge share of the web does, this is no longer a robots.txt footnote. It is a decision you have to actually make.

The three buckets, and why the split matters

Training is a crawler pulling your content to train or fine-tune a model. Your words get absorbed into the weights. You never see a referral, a click, or a sale from it.

Search is a crawler indexing your content so an AI can answer questions about it later and, ideally, point people back to you.

Agent is the interesting one. An agent is a bot fetching your page because a human asked it to. ChatGPT pulling a page to answer a live question. Claude or Gemini driving a browser to complete a task for someone. There is a person on the other end, right now, waiting.

Lumping Agent in with Training is the part builders should push back on. Training takes and gives nothing back. An agent visit is a real user, just wearing a different user agent.

robots.txt was always a polite request

For years the only lever was robots.txt, and robots.txt is a suggestion. Well-behaved crawlers respect it. The ones you actually worry about do not. Cloudflare enforcing this at the edge is the difference between a sign that says "please do not" and a locked door.

That is the real shift. Access policy for your content is moving from a text file nobody enforces to an infrastructure control that does. So the decision carries weight now.

What we would actually do

Treat it as three separate calls, not one.

• Training: block it, in most cases. There is no upside for you unless you have struck a licensing deal, and Cloudflare is building the rails to charge for that access directly.
• Search: allow it. This is discovery. Blocking it is the same mistake as blocking Googlebot in 2010.
• Agent: think hard before you block. If your customers use AI assistants to research, compare, or buy, an agent fetching your page is a channel, not an attack. For an ecommerce store this is early agentic commerce traffic. Blocking it by default means opting out of where a chunk of buying is heading.

The trap is accepting the default because it is easier. The default is tuned for ad-supported publishers protecting page views. If your business model is a product, a store, or a SaaS app, your incentives are different, and you should set your policy to match.

Check your config before September 15

If you are on Cloudflare, go look at your bot settings now, not on the 16th when an agent your customer is using gets a wall instead of your content. Map each of the three categories to an actual business decision. Write it down. Revisit it as agentic traffic grows, because it will.

We are here to help founders and teams design and build digital products that are built to scale with you, not slow you down. If you are looking to build something, get in contact with us today!