Ask an AI (or an intern, or yourself on a Monday morning) which social channel is performing better, and it's easy to get back the highest percentage in the sheet. The problem is that a percentage without its sample size behind it isn't data, it's a trap with a very convincing face. It happened to us a few days ago, with our own content dashboard, and the lesson applies to any business that looks at a report and decides something from it.
The problem: two percentages, one wrong conclusion
At AutoBoost we publish content daily across several channels (LinkedIn, Threads, Facebook, Instagram), and we wanted to know, without guessing, which one was actually sending traffic to the blog. Looking at the panel by network, Threads showed a 43.75% CTR (clicks over impressions) and LinkedIn showed 0.10%. Read that way, the decision looks obvious: put more effort into Threads, which "converts 400 times better."
That's the exact opposite of the real conclusion. Threads' 43.75% came from 16 impressions (7 clicks out of 16 times the post was seen). LinkedIn's 0.10% came from 86,721 impressions. A percentage calculated over 16 cases isn't a reliable rate, it's close to a coin flip: with that base, the result could just as easily have been 6% or 60% without anything real changing underneath. LinkedIn's number, with that much larger base, does describe real behavior, even though it looks worse at first glance.
Had we acted on the first percentage we saw, we would have bet our content effort on the wrong channel.
The trap, explained without statistics
You don't need a background in statistics to see the problem, just one question: how many cases are actually behind that percentage?
| What you see | What's behind it | Correct read |
|---|---|---|
| Threads: 43.75% CTR | 7 clicks out of 16 impressions | Base too small to decide anything |
| LinkedIn: 0.10% CTR | ~87 clicks out of 86,721 impressions | Reliable number, even if the percentage looks low |
The same pattern shows up in any business, not just social media:
- An ad campaign with an "8% conversion rate" over 12 clicks.
- A product with "double the returns" when 3 units sold versus 300.
- A salesperson with "the best close rate on the team" from 2 deals worked that month.
- A support channel with "fewer complaints" simply because almost nobody uses it yet.
In every case, the percentage is mathematically correct, and the decision made from it can still be wrong. The mistake isn't in the math, it's in comparing percentages without checking the sample size behind each one.
Why this matters more once an AI is involved
This is where the problem stops being a statistics footnote and becomes a real question about how AI gets used day to day in a business. It's increasingly common to ask an AI assistant to read a dashboard or a spreadsheet and "tell you what's going on." If you ask an AI which channel is performing better and hand it a table with these two percentages and nothing else, it will very likely come back with something like "Threads is performing much better than LinkedIn": that's the most direct reading of the data as written, and a general-purpose AI has no reason to doubt a 43.75% if nobody tells it there are only 16 cases behind it.
It's not that the AI "gets it wrong" in the sense of making a calculation error. It answers exactly what you gave it, and if the data you gave it was already framed badly, the answer inherits that mistake, and it arrives sounding like it's "been analyzed." That's exactly what makes the trap more dangerous: a badly compared percentage, said by a person, sounds like a debatable opinion; the same mistake, said by an AI in an analytical tone, sounds like a conclusion.
It's the same discipline we apply on the data and AI platform we built for a pharmacy group (you can see it in the data platform + AI case study): the AI doesn't decide based on a loose number, it reasons over data that's already validated, with its real volume behind it. The data gets ordered and put into context first, and only then does the AI draw a conclusion or make a recommendation. Do it the other way around, and it looks nice and gets the answer wrong.
The decision rule
Never compare two percentages without first comparing the size of what's behind them. In practice, here's how we apply it:
- Before looking at the percentage, look at the denominator. If one side has a volume dramatically smaller than the other (in our case, 16 versus 86,721), that side's percentage doesn't get to decide anything yet.
- If you're going to ask an AI to interpret a dashboard, give it the volume too, not just the rate. "Threads: 43.75% over 16 impressions. LinkedIn: 0.10% over 86,721" produces a completely different answer than handing over the two bare percentages.
- Set a minimum volume ahead of time, below which a percentage "doesn't count" for your business (with new channels, we wait for at least a few hundred impressions before drawing any conclusion about performance).
- When a number surprises you a lot ("400 times better"), be suspicious of the data first, don't celebrate the conclusion first. Big surprises almost always come from a small base.
Checklist before trusting a percentage (yours or your AI's)
- Do I know how many cases are behind each percentage I'm comparing?
- Do both sides have a similar volume, or is one orders of magnitude smaller?
- If I asked an AI for the read, did I give it the volume or just the rate?
- Do I have a minimum case count below which I don't draw conclusions for this type of data?
- If the result surprises me a lot, have I checked the data before acting on it?
The takeaway for any business
You don't need a data team to fall into this trap, or to avoid it. You need the habit of asking "how many cases is this actually based on?" before deciding anything, whether the number comes from a dashboard, an AI, or a report a vendor sends you. It's the exact same principle we apply at AutoBoost when we build custom software and AI: the data gets ordered and put into context first, and AI only adds real value once it reasons over data that's properly framed, not over a loose percentage.
If you're building a dashboard, an AI assistant, or any system that will make (or suggest) decisions from your data, and you want it framed correctly from day one, at AutoBoost we build that kind of custom solution, integrated into your real system. Get in touch and we'll look at it together.
