From September, Blocking AI Bots Could Hide You From Google

There is a quiet change coming to the web on 15 September, and it could help decide whether new customers can find your business at all. Cloudflare, one of the companies that sits between websites and the outside world, is changing how it treats the automated crawlers that read the web. From that date, its default settings will start blocking a whole class of crawlers on many sites unless the owner has said otherwise. If your website runs through Cloudflare, and a large share of Australian small-business sites do without their owners ever knowing, this is worth understanding now, not in September.

TechCrunch reported on 1 July 2026 that Cloudflare is rolling out new defaults aimed at what it calls mixed-use crawlers, the bots that do more than one job at once. The change applies to new customers, new websites, and every existing site on its free plan that has not adjusted its settings. Chief executive Matthew Prince framed it as protecting content, saying the company must "go further and act faster so that a sustainable ecosystem can emerge". The goal is reasonable. The side effect is where small businesses need to pay attention.

Here is the catch. The same crawler that trains an AI model often also indexes your site for Google. Cloudflare has split crawlers into three jobs, search, AI agents and AI training, but plenty of real-world crawlers do two or three of those at once. Search Engine Journal warned on 2 July 2026 that because Cloudflare applies the strictest rule, a site that blocks AI training to protect its content can end up blocking search crawlers like Googlebot at the same time. Because that happens at the network level, it is far harder to undo than a line in a robots file. Block the wrong thing, and you can quietly vanish from Google.

What is actually changing

For years the deal was simple. Let the crawlers in, get found. That deal is now being renegotiated in public. Site owners are tired of AI companies scraping their work for free, and Cloudflare's answer is to hand them a gate, and even a tollbooth: a marketplace that lets sites charge AI firms for access rather than give it away. For anyone whose content has real value, that is a genuinely new lever. But a gate you do not understand is a gate you can lock yourself out of. The businesses most exposed are the ones who never chose these settings, never knew they were behind Cloudflare, and will be shifted onto the new defaults automatically.

Why this matters more for a small business

A large publisher has a technical team watching this closely. A cafe, a trades business, a clinic or a shopfront does not. Yet the stakes are arguably higher for the small business, because so much of where your next customer comes from now runs through machines reading your site: Google's results, Google's AI Overviews, and the AI assistants people increasingly ask before they ask anyone else. If a blunt setting keeps those readers out, you do not get an error. You just slowly stop appearing, with no idea why. It is the same blind spot we looked at when we explained how most of the AI bots hitting your website are fake: the traffic on your own site is getting complicated, and guessing about it is expensive.

The people who watch this closely are already circling the theme. Aleyda Solis, one of the most respected voices in technical SEO, has spent the year pointing out that being visible in AI search is now its own discipline, with its own plumbing, separate from the old game of ranking on Google. This Cloudflare change is exactly that plumbing: dull, invisible, and decisive. Get it right and you stay open to the readers that bring customers while closing the door on the ones that only take. Get it wrong and you are invisible to both.

Where the opportunity is

Handled properly, this is not a threat at all. It is a chance to be deliberate about something most of your competitors will get wrong or ignore. Here is what good looks like once your site's crawler access is set up for how search and AI actually work today:

• Google and Bing can still read and index you, so you keep the search visibility you have already earned.
• The AI assistants your customers ask can still find and cite you, so you show up in AI answers, not only the blue links.
• The crawlers that just strip your content for free can be turned away, or sent to the tollbooth, on your terms rather than theirs.
• You move onto the new defaults on purpose, with settings you chose, instead of waking up on 15 September to whatever a switch flipped to.
• Someone is watching for the next change like this one, because there will be a next one.

The web is quietly deciding who gets to read your website. The businesses that stay visible will be the ones who made that choice on purpose.NextAura

You do not need to become an expert in crawler policy, and you certainly should not go poking at settings you are unsure of, since that is the quickest way to take yourself offline. The point is simply this: a real change lands on 15 September, it is easy to get wrong, and the cost of getting it wrong is invisibility in exactly the places your customers now look. This is the sort of unglamorous, high-stakes work we handle at NextAura. We make sure your AI search and SEO visibility is built so the right readers get in and the freeloaders do not, and we stay across changes like this one so you never have to. Get in touch and let us keep you visible while you get back to running the business.