The AI Visibility Score You're Being Sold Is Mostly Noise

A new wave of tools promises to tell you exactly how visible your business is inside AI answers. A single score. A neat leaderboard. A number that moved up or down since last month, ready to screenshot into a report. If you have been anxious about whether ChatGPT, Gemini or Google's AI answers ever mention your business, that tidy number is enormously reassuring. New research suggests most of those numbers are close to meaningless.

Two independent pieces of research landed this month pointing at the same problem. A forthcoming preprint from the analytics firm IQRush, led by co-founder Ron Sielinski, and a separate April 2026 study from researchers at the University of St. Gallen, both concluded that a single measurement of how often an AI cites your brand is mostly statistical noise. The finding was written up by Search Engine Journal on 11 July 2026, and it deserves the attention of anyone who has been quoted a price for an AI visibility dashboard.

This matters because the money is already moving. Generative engine optimisation, or GEO, the work of getting your business named and recommended inside AI answers, is one of the fastest-growing lines in small business marketing budgets. And where budgets go, dashboards follow. The risk is that owners start steering real decisions off a figure that is far shakier than it looks.

Why one snapshot cannot be trusted

Generative models do not answer the same question the same way twice. Ask an AI the same thing on a Monday and again on a Friday and it may cite different sources, in a different order, for reasons that have nothing to do with your website. That randomness is designed into how these systems generate language. It is a feature, not a glitch, and it means any single reading of who an AI cites is one roll of the dice, not a fixed fact.

The research puts numbers on how much this distorts a ranking. In one test reported from the work, a query about running gear on an AI search product credited one publisher with roughly 9.5 per cent of citations and another with about 6.0 per cent. On paper the first is clearly winning. But the 3.5 point gap between them sat inside the margin of error, which means a single reading genuinely cannot tell you which one the AI prefers. A dashboard would have shown a confident leader that the underlying data does not support.

The real measurement is bigger than the dashboard admits

The researchers also measured how much sampling it actually takes before a ranking stops jumping around. Across 30 platform and topic tests, the number of cited answers needed before the order settled ranged from 33 to as many as 94. Three of the 30 never stabilised at all, even after 125 questions. One topic needed 104 usable answers out of 125 attempts, because the rest cited nothing at all. A tool that reads you a confident score off a handful of prompts is not measuring your visibility. It is guessing, with a nice interface wrapped around it.

Rigorous voices in the field have been saying a version of this for a while. Mike King, one of the more measurement-minded figures in generative search, has long argued that AI visibility has to be handled as a data problem, not a scoreboard. This research is the clearest evidence yet that he is right: to get a number you can trust, you need enough samples that the model's natural randomness averages out, and you need the gap between you and a rival to be bigger than the margin of error before you claim you are ahead.

What this means for a small business

For an owner, the temptation is obvious and it is expensive. You see your score dip, you assume you are losing ground, you rewrite content or shift spend to chase it back up, and next month it moves again on pure chance. You have paid, in time and money, to sprint on a treadmill. The most damaging thing about a bad measurement is not that it is useless, it is that it is convincing enough to act on. Acting on noise is worse than having no number at all, because at least an honest blank does not send you in the wrong direction.

None of this means AI visibility does not matter. It matters more every month, and being recommended inside an AI answer is quickly becoming as valuable as ranking on the first page of Google once was. It means the measurement is harder than a one-line score pretends, and separating a real change from everyday randomness is exactly the sort of fiddly, easy-to-get-wrong work that pays to hand to someone who does it properly. Handled well, this is what good looks like:

• You are told the truth about how visible you are, with enough sampling behind the number that it actually means something, not a confident figure pulled from a few prompts.
• Real movement is separated from everyday randomness, so you never spend money chasing a dip that was only ever noise.
• Measurement is tied to the outcomes that pay you, appearing when a customer asks, being recommended, winning the click or the call, not a vanity leaderboard position.
• You know which changes genuinely moved the needle, because they were tested against the model's natural variation rather than a single before-and-after screenshot.
• Effort goes where it compounds, so your presence in AI answers grows steadily instead of bouncing around a dashboard every month.

A single AI visibility score is a snapshot of a moving target. Treated as fact, it will send you chasing noise. Measured properly, it becomes something you can actually act on.

The takeaway for a small business owner being pitched an AI visibility tool is not to ignore the space. It is to ask a harder question before you buy: how many answers is this number built on, and is the gap it is showing me bigger than the margin of error? If the answer is a shrug, the score is decoration. Getting AI search right is real, valuable work, and the first step is refusing to make decisions off a measurement that cannot survive being asked twice.

This is exactly the kind of work we do at NextAura. We track how Australian small businesses show up across AI search and generative engines, and we do it with the sampling and the statistical care that turns a noisy number into a decision you can trust. If you would rather not gamble your marketing budget on a dashboard that changes its mind every month, and you want the same clear-eyed read we brought to why search demand is at a record high, not dying, get in touch and we will measure it properly and move it in the right direction while you get back to running the business.