Why one AI answer proves nothing: ask three times, on three days
Ask an AI assistant the same buyer question twice and you will often get two different lists of companies. Here is how often, measured on our own 1,800 answers, and what that means for anyone counting AI visibility.
Measured 21 September – 3 October 2026 · 600 question series, 1,800 answers · Satayu Isariyaphorn
A single AI answer is one draw from a distribution, not a fact about the assistant. In our own data, when the same question was asked to the same assistant on two separate days, the set of companies it named was identical only 55% of the time. Across three days it was identical in 44% of series. And in 40% of the series where a company was named at all, it was named on only one or two of the three days, so a single run would have called it visible or invisible depending on which day you happened to ask. That is why every figure we publish comes from three runs on three separate days, with the sample size beside it.
What we measured
Between 21 September and 3 October 2026 we asked ChatGPT, Claude, Perplexity and Google’s AI Mode a fixed set of buyer questions for 22 B2B software companies (five questions each) and for ourselves (forty questions). Every question went to every assistant three times, once a day on three separate days, each time in a fresh stateless request with no history, no account and no follow-up. That is 600 question-and-assistant series and 1,800 answers. For each answer we recorded which of the tracked companies (the company measured and its named rivals) appeared, under counting rules written before collection.
“Same list” below means the same set of tracked companies, ignoring order. It is a lenient test: two answers can differ in every other sentence and still count as the same here.
How often two days agree
| Assistant | Day-pairs with the identical set | Share |
|---|---|---|
| Perplexity | 307 of 450 | 68% |
| ChatGPT | 293 of 450 | 65% |
| Claude | 201 of 450 | 45% |
| Google AI Mode | 181 of 450 | 40% |
Google’s AI Mode and Claude vary the most, which fits how they work: both search the web for most buyer questions and build the answer from what they find that day. Perplexity and ChatGPT returned the same set of tracked companies about two times in three.
What one run gets wrong
The question a buyer cares about is simpler than the whole list: was my company named? Across the 600 series, the company measured was named on all three days in 156, on two days in 51, on one day in 55, and on none in 338.
| Named on | Series | What a single run would have told you |
|---|---|---|
| 3 of 3 days | 156 | Right whichever day you asked |
| 2 of 3 days | 51 | “Not named” one day in three |
| 1 of 3 days | 55 | “Named” one day in three |
| 0 of 3 days | 338 | Right whichever day you asked |
So 106 of the 262 series in which the company was named at all (40%) were mixed. If you check once, screenshot the answer and conclude “ChatGPT recommends us” or “it doesn’t”, in those series you are reporting the weather on the day you looked.
Others have found the same thing, at larger scale
SparkToro ran 2,961 prompts across ChatGPT, Claude and Google’s AI in January 2026 and found that the chance of two runs of the same prompt returning the identical list of brands was under 1 in 100 (their write-up). Their test is stricter than ours — the whole list of brands, not just the tracked ones — which is one reason their number is lower. The conclusion is the same: one answer is not a measurement.
What to do about it
- Freeze the wording. Write the questions down and never change them. A reworded question is a new measurement, not a change in the market.
- Ask cold. Fresh session, no history, no account, no follow-up. Your own chat history and memory change what you see.
- Three runs, three days. Three answers an hour apart are closer to one answer than to three. Spread them over separate days so you see the variation instead of one draw from it.
- Report the fraction, with the denominator. “Named in 7 of 12 answers” is a measurement. “ChatGPT recommends us” is a screenshot.
- Compare months under the same rules. The number only means something when September and October were taken the same way.
Check the figures
The counting rules, the questions and the scripts are public on GitHub under CC BY 4.0. The consistency figures on this page come from consistency.py run over our own collected answers; the per-company reports they come from are the ones we send to each company free. If you take your own measurement and get a different consistency rate, we would like to see it.