Lies, damned lies, and AI visibility statistics

"If AI answers the question, your 'traffic strategy' isn't a strategy."

That's the hook everyone nods at now - because it's true. Buyers are getting decisions from AI surfaces, and your website is increasingly proof, not discovery.

But there's a second, quieter truth that's about to cause just as much damage:

AI visibility measurement is entering its "lies, damned lies, and statistics" era.

Not because the people doing the research are dishonest. Because the incentives are obvious: if you can make AI look wildly inconsistent, you can sell more "monitoring". If you can make it look stable, you can sell more "optimisation". If you can make it look scientific, you can sell both.

And with the right statistic, you can "prove" almost anything.

Where the phrase comes from (and why it matters here)

"Lies, damned lies, and statistics" is one of the best-known jabs at how numbers can be used to launder weak arguments into something that looks authoritative. It was popularised in the US by Mark Twain, who attributed it to Benjamin Disraeli - though the attribution is disputed and earlier versions appear in late-19th-century British sources.

That's the point: the quote itself has messy provenance. Which is fitting, because this entire debate is about messy measurement being presented as certainty.

The SparkToro-style claim: "AI is so inconsistent you need hundreds of repeats"

You've likely seen the headline version:

• Ask the exact same question repeatedly.
• Get different brand lists.
• Therefore you need lots and lots of runs for statistical confidence.
• Therefore, if a tool isn't making enough calls, the data is unreliable.

On the surface, it sounds sensible. Variance exists. Small samples mislead. Statistics fixes that.

But this framing hides a sleight of hand:

It defines "consistency" in a way buyers don't actually experience

In research from Gumshoe, "consistency" is effectively:

"Do I get the exact same list of brands in the exact same order?"

That is a very strict definition - and it's designed to fail, because AI systems aren't built to be deterministic in that way. Even when the "intent" is stable, outputs vary due to retrieval differences, model updates, context window effects, and plain old stochasticity.

So yes - if your success condition is "identical list, identical order", you can make the required sample size look terrifying.

That's not measurement. That's theatre.

The uncomfortable counterpoint: exact prompt wording is the wrong unit of analysis

Here's what's misleading in the "run the same prompt 124 times" narrative:

Buyers don't run prompts like lab rats

Real people don't ask the same question 124 times. They ask variations:

• "Best tools for X"
• "Top providers for Y"
• "Alternatives to Z"
• "Which vendor is good for [constraint]?"
• "Is [vendor] safe/credible/enterprise-ready?"

Different wording, same intent. And LLMs optimise for intent far more than phrasing.

AI visibility isn't "did the output match character-for-character"

The metric that matters commercially is closer to:

• Are we mentioned for the right intents?
• Are we recommended or merely listed?
• Is our positioning accurate?
• Are we cited, and are the citations credible?
• Are we compared fairly vs the obvious competitors?
• What's the sentiment / framing?

You can get stable signal here across a well-designed prompt set - even if individual responses vary.

In other words: variance at the prompt-output level does not automatically imply variance at the decision level.

You can "prove" instability by choosing an unstable metric

This is the statistical trap:

If you measure something hyperspecific (exact ordering of brands), you'll observe high variability and conclude "we need more data".

If you measure something aligned to intent (share of voice, recommendation rate, citation rate, negative vs positive framing), you'll often find that you don't need anywhere near that volume to make useful decisions - especially when you aggregate across prompts and engines.

Both are "true" within their definitions.

Only one helps a CMO decide what to ship next week.

Another truth: the models are changing faster than your confidence interval

Even if the maths is impeccable, the system you're measuring is not stationary.

Models update. Retrieval sources shift. Tooling changes. Safety and ranking policies are tweaked. What was "true" last month can be untrue next week.

So there's a danger in treating AI visibility like classic SEO rank tracking - where the environment changes, but not this quickly.

The result is a perverse outcome:

• You can spend a fortune achieving statistical certainty…
• about something that no longer exists.

This doesn't mean "don't measure". It means measure with a cadence and method that matches the volatility.

A better way to measure AI visibility (that doesn't collapse under its own maths)

If you want measurement that's both rigorous and commercially meaningful, design it around intent:

1) Use a prompt library, not a single prompt

Build 50–150 prompts that represent real buyer intents:

• discovery
• shortlist
• comparison
• verification
• objections (security, pricing, implementation, compliance)

2) Aggregate into decision metrics

Track:

• answer share-of-voice (mentions across prompts/engines)
• recommendation share (explicit "I recommend X")
• citation rate (and which sources)
• framing/sentiment (how you're described)
• competitor displacement (who wins when you lose)

3) Treat time as a first-class variable

Re-run weekly (or even more frequently for high-stakes categories) and show trends, not point estimates.

4) Be honest about what you can and can't claim

You can be confident about directional visibility and comparative positioning across intents - without pretending you can predict the exact brand order on a single prompt tomorrow afternoon.

That's "statistics" used properly: to reduce uncertainty about the thing you actually care about.

The real risk: marketers optimising for the wrong certainty

AI has moved discovery upstream. Your website is proof, not the starting gun.

And now measurement is moving too.

The danger isn't that people will stop measuring. It's that they'll measure the wrong thing - then confidently optimise the wrong work.

So yes: be sceptical when someone tells you they've solved AI visibility with a single number.

And be just as sceptical when someone uses scary-sounding sample sizes to imply that nothing is knowable unless you buy their platform.

That's the oldest trick in the book.

Just… with a new surface.

Want to test this properly on your brand?

If you want to see what AI actually says about you across buyer intents - and what to change to improve it - you can run it yourself.

Sign up for a free account and get 25 answers: