“Cloudflare are going to by default block all AI crawlers aren’t they?”

I’ve had that question three times this month, and althought it’s a fair question, this is really indicative of the scaremongering that exists in our industry currently. There are some important checks we should all be doing in CloudFlare, but keep calm. It’s really not as seismic as many are making out.

Cloudflare is not blocking all AI crawlers on 15 September 2026. What it is doing is more specific, affects fewer sites than the headlines imply, and carries a different risk that the one being reported. If someone on your team ticked a box in 2025, your site could potentially drop out of Google search results. But, the chances of that are low, and easily rectified.

Here’s what actually changes, who it applies to, and the ten minute checks worth performing before September 15th 2026.

TL:DR – What Are The Upcoming CloudFlare Changes

• Cloudflare has replaced its single “Block AI Bots” switch with three separately controllable behaviours. Search, Agent, and Training. These went live on 1 July 2026 for every customer, including the Free tier.
• On 15 September 2026, new defaults block Training and Agent bots on pages that display ads. Search stays allowed by default. This applies to new domains, new sites added by existing customers, and existing free tier customers who haven’t touched their settings.
• Also on 15 September, multi-purpose crawlers get judged on all of their behaviours, with the most restrictive rule winning. Googlebot, Applebot, and Bingbot crawl for search and training in a single bot, so blocking Training blocks them too.
• That second change applies to everyone, regardless of plan or how long the domain has been on Cloudflare.
• You can opt out in your Cloudflare Security settings at any point before 15th September.

What Cloudflare actually announced

On 1 July 2026, its second “Content Independence Day“, Cloudflare scrapped the binary approach to AI bots. Instead of asking whether a crawler is AI or not, it now asks what the crawler does with your content once it has it.

That produces three categories you can allow or block independently.

This taxonomy is the useful bit, and it’s the part that will outlive the September deadline. Cloudflare is also tracking Transact, Data Collection, Security Testing, SEO, Ads Verification, Social, Feed Fetching, and Monitoring behaviours, with the three above exposed as configurable options to all customers.

Two separate things happen on 15 September

Almost all of the confusion in the coverage comes from collapsing these into one headline. They’re different changes with different scope.

1. New defaults on ad-monetised pages

Training and Agent bots get blocked by default on pages that display ads. Search remains allowed. Cloudflare’s reasoning is that an ad signals the page was built for a human to land on, so bots that divert that attention, or absorb the content without returning anything, get kept out.

Scope matters here. This applies to new domains onboarding to Cloudflare, new sites added by existing customers, and all existing Free tier customers who haven’t changed their settings by the deadline. An established paid zone with configured bot settings does not get swept into it.

If your site carries no advertising, this change is largely academic for you.

2. Multi-purpose crawlers get judged on everything they do

This is the one that matters, and it applies to everybody.

From 15 September, a crawler that combines Search with Training is allowed or blocked according to all of its behaviours, and the most restrictive applicable rule wins. Googlebot, Applebot, and Bingbot all crawl for search indexing and model training using one bot. Cloudflare names all three explicitly.

So if you have Training blocked, whether through the new controls or the legacy “Block AI bots” preset, Googlebot is caught in that block from 15 September.

Why this is a Google problem, not an AI visibility problem

Every other write-up I’ve read has framed this as an AI visibility deadline. I’d frame it the other way round, or perhaps more of an AIR SEO consideration.

A Cloudflare block operates at network level. It isn’t a robots.txt directive expressing a preference that a crawler can weigh up and ignore. The request doesn’t reach your server. If Googlebot is blocked, Google cannot crawl you, and a site Google cannot crawl does not stay in the index.

The furore from Mid 2025, when everyone was panicking about AI scraping, has most probably caused this issue and the uncertainty. Developers, IT managers and well-meaning marketers were most likely logging into Cloudflare and ticking “Block AI Bots”. Job done. Hmmm, as an SEO, I never felt compelled to do this, but each to their own.

In some cases, that person has likely since left, or maybe even forgotten. On 15th September the classification logic changes underneath a setting nobody has looked at in a year, and that’s the bit that poses a risk to your organic traffic.

CloudFlare checks to run before 15th September 2026

This takes minutes, not hours. Do it for every client site, not just the ones you think are on Cloudflare (as there are other ways to block bots and agents).

• Confirm whether the site is behind Cloudflare at all. Plenty of agencies bundle it into hosting without telling the client. Check the nameservers or response headers, or just ask whoever built the site.

• Check the plan tier. Free zones are the ones exposed to the new defaults sweep. Paid zones with configured settings are not.

• Open Security settings and look at the AI bot configuration. You are hunting for two things. The legacy “Block AI bots” preset, and any block on the Training category. Either one catches Googlebot.

• Screenshot what you find. Settings change quietly. Forward these to the relevant team to rectification, or implement yourself if you have access.

• Use the opt-out if the client genuinely wants Training blocked. Cloudflare lets you opt out of the new default configuration in Security settings, which confirms no change to Training crawlers that also crawl for Search. That preserves the Training block without taking Googlebot down with it.

• Get server logs. This is the step everyone skips and then regrets. Logs are the only way to know which bots actually reach your origin, how often, and what they get served. If you don’t have log access, request it now, because in my experience that ticket takes weeks to land (if at all, but that’s a separate conversation).

Allow the agents. Seriously.

Search should stay allowed. In my humble opinion.

Agent is the category worth actual thought, and my view is that most businesses should allow it. An Agent request means a real person has asked an assistant a question and that assistant is fetching your page to answer it. Blocking Agent is blocking a customer mid-research. It’s the 2026 version of blocking your own shopfront and wondering where the footfall went.

Training is the only genuinely commercial decision in the set, and it’s the one to make deliberately rather than by default. Just understand what blocking it now costs you, because from 15th September it costs you Googlebot hits unless you opt out.

The bigger shift, which isn’t about September at all

Two changes in this announcement will matter long after the deadline passes.

First, Verified no longer means allowed. Previously every Verified bot was allowed by default. Now Verified only makes a bot allowable within its category, and the categories you permit decide what gets through. Bot access has stopped being a switch and become a policy.

Second, Cloudflare is testing a use signal that extends Content Signals in robots.txt, with three values:

• immediate (interact but store nothing),
• reference (index, excerpt, and link back)
• full (summarise and reproduce)
• Managed robots.txt now gets use=reference appended automatically.

That’s a preference rather than a block, but it’s the first serious attempt at expressing how content may be reused rather than simply whether it can be fetched.

This is why I keep arguing that AI search is a retrieval layer sitting on top of organic search, not a separate channel with its own playbook. Crawl access, render completeness, and whether your content survives retrieval are the same fundamentals they’ve always been. The controls just got more granular, and the cost of getting them wrong went up.

You don’t need a GEO strategy for this. You just need to know what’s in your Cloudflare settings and consider search holistically.

Some quick questions about CloudFlare settings and SEO

Is Cloudflare blocking all AI crawlers on 15 September 2026?

No. Cloudflare is changing default settings for two of three bot categories on specific types of page. Training and Agent bots are blocked by default on pages that display ads, while Search bots remain allowed. Existing paid customers with configured settings are not moved to these defaults at all.

Does this affect every website on Cloudflare?

No. The new ad-page defaults apply to new domains, new sites added by existing customers, and existing Free tier customers who have not changed their settings. The multi-purpose crawler change is different: it applies to every customer, on every plan, from 15 September.

Can Cloudflare block Googlebot?

Yes, and this is the main risk. Googlebot crawls for both search indexing and AI training using one bot. From 15 September, multi-purpose crawlers are evaluated against all of their behaviours, with the most restrictive rule applied. Any site that blocks the Training category, including through the legacy “Block AI bots” preset, will block Googlebot as well. Applebot and Bingbot are affected the same way.

What happens if Googlebot is blocked at Cloudflare?

The request never reaches your server, so Google cannot crawl or refresh your pages. Over time, pages drop out of the index and organic traffic collapses. Unlike a robots.txt directive, a Cloudflare block operates at network level and cannot be ignored by the crawler.

How do I opt out of the new Cloudflare defaults?

Open your Cloudflare dashboard, go to the zone’s Security settings, and mark the opt-out at any point before 15 September 2026. This confirms you want no change to Training crawlers that also crawl for Search, which keeps Googlebot allowed while preserving your Training block.

Should I block ChatGPT and other AI agents from my website?

For most businesses, no. Agent bots such as ChatGPT-User visit because a real person has asked an assistant a question and is waiting on the answer. Blocking them removes you from the research a potential customer is doing right now. Training crawlers are a separate commercial decision worth making deliberately.

What is the difference between Search, Agent, and Training bots?

Search bots crawl to build an index they will later use to answer questions, and typically return referral traffic. Agent bots visit in real time on behalf of a person who is waiting for an answer. Training bots take content to train or fine-tune a model, which absorbs it permanently with nothing sent back. Cloudflare lets you allow or block each independently.

How do I check my Cloudflare AI bot settings?

Log in to the Cloudflare dashboard, select the domain, and open Security then Settings. Look for the AI bot management options and check whether the legacy “Block AI bots” preset is enabled, and whether the Search, Agent, and Training categories have been configured. If Training is blocked and you have not opted out, Googlebot will be blocked from 15 September.

Need someone to sanity check your bot access before the deadline? That’s one of the many things I do. Get in touch.