If your site sits behind Cloudflare and you’ve never touched its AI bot settings, there’s a real deadline in about two weeks worth checking before it flips a default for you.
On July 1, 2026, Cloudflare announced a change to how it handles AI crawler traffic across a large share of the web. The new defaults take effect September 15, 2026.
What’s actually changing.
Cloudflare now sorts AI crawler traffic into three categories: Search crawlers (index content to answer questions later, includes traditional search engines and AI systems that credit or link back), Agent crawlers (act in real time on someone’s behalf, like a chat assistant fetching a page while answering a question), and Training crawlers (pull content to train AI models, and this bucket also catches any crawler that doesn’t clearly declare which of the three it is).
Starting September 15, Training and Agent crawlers get blocked by default on any page Cloudflare’s system classifies as showing ads. Search crawlers stay allowed by default. This new default applies automatically to new Cloudflare customers, new sites added by existing customers, and , importantly, every existing free-tier customer. Paid customers with settings already configured keep what they have.
Here’s the honest part: it probably doesn’t touch you directly.
The new default only fires on pages that actually display ads. A typical business website a real estate agency, a clinic, a restaurant, a retail site, the kind of site hexweb.dev builds, doesn’t run ad inventory, so this specific default change has nothing to flip for most business owners reading this.
But it’s still worth checking your settings anyway
Here’s why this is genuinely relevant regardless: Cloudflare’s dashboard still has an older, simpler “block all AI bots” toggle sitting alongside the newer granular controls. If that legacy switch is turned on something a site owner might have flipped on months ago without thinking much about it, it doesn’t just block AI training crawlers. It can block the exact Search-category crawlers that AI systems use to find and cite your content in things like Google AI Overviews and AI Mode, the same systems this section has covered repeatedly. A site actively trying to be found and cited by AI search, while accidentally blocking the crawlers that make that possible, is a real and easy mistake to make without ever realizing it.
Misconfigured crawler settings aren’t a hypothetical either there have been real cases this year of businesses losing their search visibility for weeks because a Cloudflare setting was blocking Google’s crawler entirely, without anyone noticing until traffic had already dropped.
What to actually do
If your site is behind Cloudflare: log into your dashboard, go to Security Settings, and look for AI bot policy controls. Confirm the legacy “block all AI bots” toggle isn’t silently switched on, and if you do want fine-grained control, the newer three-category system lets you allow Search crawlers while still blocking Training or Agent traffic specifically.
Practical takeaway: the September 15 deadline itself likely doesn’t require action for a typical business site but checking that nothing is accidentally blocking the crawlers your AI visibility depends on is worth five minutes regardless of the deadline.
Not sure what your own site’s crawler settings actually allow right now?