Cloudflare has rolled out a major update to how it manages bots, replacing its old “block all AI bots” switch with granular, category-based controls. If your website runs on Cloudflare, this update changes which crawlers can reach your content by default from 15 September 2026, and that has real consequences for your visibility in both Google and AI search. Here is what the Cloudflare Bot Management update includes and what to do about it.
Quick answer: Cloudflare now sorts AI crawlers into three categories, Search, Agent and Training, and from 15 September 2026 it blocks Training and Agent bots by default on ad-monetised pages while keeping Search allowed. The update also adds BotBase (a searchable bot database) and Content Use Controls. If AI and search visibility matter to you, keep Search crawlers allowed and review your settings before the deadline.
What is Cloudflare Bot Management?
Cloudflare sits in front of millions of websites and can identify and control the automated traffic, or bots, that visit them. Bot Management is the set of tools that lets a site owner decide which bots to allow, challenge or block, from search engine crawlers to scrapers and AI agents. This update overhauls how those controls work for AI crawlers specifically.
What is changing in the 2026 update
Cloudflare previously offered a single option to block AI bots outright. It now replaces that with granular controls based on how a bot uses your content:
|
Crawler type |
What it does |
Default from 15 Sep 2026 |
|
Search |
Indexes pages for search and cited AI answers |
Allowed |
|
Agent |
Acts on behalf of a user in real time, for example an AI assistant |
Blocked on ad-monetised pages |
|
Training |
Collects content to train AI models |
Blocked on ad-monetised pages |
Cloudflare’s logic is that an ad signals a page meant for a human to see, so on those pages it keeps human attention as the goal and holds back bots that do not send visitors. The controls are available on all plans, including the Free plan, and you can customise or opt out in your settings before the deadline.
Important: the policy affects major multi-purpose crawlers too, including Googlebot, Applebot and Bingbot, because of how mixed-use bots are handled (see below).
New features in this update
BotBase: a searchable bot database
Cloudflare has added BotBase, a searchable catalogue of known bots, including verified bots and AI agents. Enterprise Bot Management customers get a centralised view to browse Cloudflare’s bot catalogue, search specific bots, see their classifications, filter traffic by individual bot, and copy detection IDs for use in security rules. It makes it far easier to see exactly who is crawling your site and why.
Content Use Controls
For Enterprise Bot Management users, Cloudflare is introducing content use controls that specify what a crawler may do with your content after crawling it, across three tiers:
|
Tier |
What it permits |
|
Immediate |
Use now, with no storage or reuse |
|
Reference |
Indexing, excerpts and backlinks |
|
Full |
Summaries or full reproduction |
This gives publishers a clearer say in whether their content can be stored, quoted or reproduced by AI systems.
The mixed-use trap
Around a third of crawler activity now comes from bots that blend purposes, for example a bot that does both search and training. Cloudflare applies the most restrictive rule to these. So if you block Training, a multi-purpose crawler, potentially even Googlebot on ad-monetised pages, can be blocked too. This is the single biggest risk in the update: an aggressive block can accidentally hurt your normal search visibility.
Why this update matters for your SEO
Bot controls are no longer just a security setting. They now decide who can read your content, which directly affects visibility.
- Blocking everything hurts you. Blanket-blocking AI crawlers removes your content from the AI tools your customers increasingly use to research and buy.
- Search crawlers are the ones you usually want. They feed search results and cited AI answers that can drive traffic. Cloudflare keeps these allowed by default, which suits an SEO and AI SEO strategy.
- Agent bots are a judgement call. As AI assistants that browse and buy on behalf of users grow, blocking them protects content but may cost future visibility, especially for ecommerce.
- Training bots give little back. They do not cite or link to you, so blocking them is a reasonable way to protect content.
What to do before 15 September 2026
- Open your Cloudflare dashboard and find the AI crawler or Bot Management controls.
- Keep Search crawlers allowed to protect your presence in search and AI answers.
- Decide on Agent crawlers by business type. Ecommerce and lead-gen sites may want to allow them; content sites protecting ad revenue may block them.
- Choose your Training stance. Block to protect content, or allow if you want to be in training data.
- Watch the mixed-use risk. Confirm that blocking Training does not unintentionally block Googlebot or Bingbot on your ad pages.
- Use BotBase to see exactly which bots hit your site and adjust rules.
- Verify with your logs that the bots you want are getting through, and monitor after any change.
Frequently asked questions
What is the Cloudflare Bot Management update?
It replaces Cloudflare’s single “block AI bots” option with granular controls that sort crawlers into Search, Agent and Training, plus new BotBase and Content Use Controls, with new default settings from 15 September 2026.
Will this affect my Google rankings?
It can. Multi-purpose crawlers like Googlebot can be caught by a Training block on ad-monetised pages. Check your settings so search access stays open.
Should I block AI bots on my website?
Not blindly. Keep Search crawlers allowed to stay visible in search and AI answers. Treat Agent and Training bots as deliberate choices.
What is BotBase?
A searchable database of known bots in Cloudflare, letting Enterprise Bot Management users view classifications, filter traffic by bot and build security rules more easily.
Do I need to do anything if I am happy with my setup?
You can opt out of the new defaults in your Cloudflare settings before 15 September 2026 to keep things as they are.
Get your site ready for AI and search
Crawler controls are one piece of a bigger shift toward AI-driven search. If you want your content to rank and be cited across Google and AI tools, MediaPlus can help. Explore our SEO services in Singapore and AI SEO and GEO services, and make sure your site’s technical SEO foundations are solid, or talk to our team for an AI-ready SEO plan.
