← Back to blog
Hot take: the change everyone is posting about probably won’t touch your site at all. A different line in the same announcement almost certainly will, and it’s the one that can quietly pull you out of Google.
On July 1, Cloudflare published new controls that split automated traffic into three categories and set a date for new defaults. Since then the trade press has run some version of the same headline: AI crawlers blocked by default on September 15. If you run marketing for a B2B company, that headline is mostly about somebody else’s business model.
Here’s what the announcement actually says, and what your team should check on because of it.
What changed on September 15
Cloudflare’s post lays out two separate changes that happen to share a date.
- For all new domains onboarding to Cloudflare, the Training and Agent categories will be blocked by default on pages that display ads. Search stays allowed by default.
- Multi-purpose crawlers, meaning the ones that combine Search with Training, will be allowed or blocked according to all of their behaviors, with the most restrictive applicable rule winning.
The first one is a publisher story. Cloudflare’s reasoning is that an ad on a page is a signal the owner wanted a human to land there and see it, so bots that interrupt that attention get held back while the ones that send visitors get through.
Your pricing page doesn’t run display ads. Neither does your case study library. If your site is ad-free, which describes almost every mid-market and enterprise B2B site, that default doesn’t apply to a single URL you own.
If someone on your team flipped the “Block AI bots” switch in 2024, September 15 was the day that decision starts costing you Googlebot.
The toggle you forgot you turned on
Cloudflare names three crawlers explicitly: Googlebot, Applebot and BingBot. All three index for search and feed AI products from the same crawl, which makes them multi-purpose under the new taxonomy.
From September 15, if you’ve selected to block Training, those crawlers get blocked too. That applies whether you set it through the new controls or through the legacy “Block AI bots” service that’s been sitting in Cloudflare dashboards since 2024. Plenty of marketing teams turned that on during the first wave of AI scraping panic, told the CMO it was handled, and never looked at it again.
Cloudflare sits in front of more than 20% of web domains, so the number of sites carrying a two-year-old toggle nobody has revisited is not small.
You can opt out. Cloudflare says site owners can mark this in their zone Security settings any time, which confirms you want no changes to Training crawlers that also crawl for Search. That’s a five-minute job for whoever owns your DNS.
Search, Agent, and Training
The taxonomy is the foundational part of the update. Cloudflare’s definitions:
For a B2B marketing site, those three carry very different weight.
Search you want, obviously. Training is the genuine judgment call, and reasonable teams land in different places on it. Agent is the category most marketing teams haven’t thought about yet, and it’s growing fastest.
Consider what Google described at I/O in May. AI Mode passed a billion monthly users within a year of launch, with queries more than doubling every quarter since. Google also announced information agents that run in the background around the clock, scanning blogs, news sites and social posts for changes relevant to a user’s standing question.
An agent evaluating vendors on behalf of a procurement lead is a buyer. It arrives with a human waiting on the other end of it. Blocking that traffic to protect your content from training is a trade most B2B teams would not make on purpose.
Content use, and the setting under the setting
Cloudflare also introduced a second dimension: what a bot can keep and reshare after it crawls you. Three levels, from least to most permissive:
Immediate, meaning interact but store and reuse nothing
Reference, the default, meaning index, excerpt and link back
Full, meaning summarize and reproduce
There’s a matching signal now being tested in robots.txt, extending Content Signals with a fourth optional field. Customers already using Cloudflare’s managed robots.txt get use=reference added automatically, producing something like this:
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
For most B2B marketing sites, reference is the right answer and it’s already the default. You want your positioning excerpted and linked. What you don’t want is a model reproducing your gated research in full to somebody who never visits.
Worth noting that these are preferences, not enforcement. Cloudflare is clear that robots.txt values signal intent rather than issuing blocks. Enforcement happens at the network layer
Robots.txt tells well-behaved bots what you’d prefer. Your CDN settings decide what actually gets through. Teams keep confusing the two, then wonder why their traffic doesn’t match their policy.
Where this meets how your site is built
None of the above is a CMS problem. It sits at the network layer, and you’d run the same audit on a WordPress site, a Contentful build or a static Astro site.
Where architecture does matter is what happens after you’ve made the decision. Sites that render content server-side and ship clean, structured markup give search crawlers and agents something to work with. Sites that hide their content behind client-side rendering, or bury it in markup no parser can follow, are hard to read regardless of which categories they allow. We’ve seen both, often at the same company on different subdomains.
Access control and readability are two different jobs. September 15 was a deadline on the first one, but the second one has no deadline, which is exactly why it keeps sliding.
If you want a second set of eyes on either, we’re happy to take a look. Let’s Connect.
contact us
