Bias in data analysis: 6 traps that distort decisions

Your dashboard never lies. The question you asked it does. Biases in data analysis almost never show up as a formula error. They show up as a conclusion that looked obvious, got presented at Monday’s meeting and became budget by Tuesday.

More data has made this worse, not better. Any manager can now open GA4, Meta Ads and the CRM and find, in ten minutes, a number that supports the idea they had before opening any of them. That is not analysis. It is shopping for ammunition.

The good news: the biases that cost companies the most money are few, they repeat constantly, and each one can be defused with a simple question asked before the decision is final.

What are biases in data analysis?

Biases in data analysis are systematic distortions that lead to wrong conclusions even when the numbers are correct. They come from three places: data that arrives incomplete (as in survivorship bias), the way we choose what to look at (confirmation bias), and hasty readings of how variables relate (mistaking correlation for causation).

Note how this differs from a data quality problem. There, the number is wrong: a duplicated event, a broken UTM, a sale recorded twice. Here, the number is right and the conclusion is wrong. That is more dangerous, because it passes every technical audit.

Why bias grows when a company becomes “data driven”

It sounds contradictory, but it makes sense. Once data becomes political currency inside a company, people start using it to win arguments instead of to find out what is going on.

A Salesforce survey published in April 2025, covering 552 decision makers at U.S. companies with more than 500 employees, found that 76% feel increasing pressure to back up their arguments with data, and 57% say they compete with colleagues to prove their value with data (Salesforce, 2025). In the same survey, confidence in data accuracy fell 27% compared with 2023.

Combine pressure to prove a point with low trust in the source and you get the perfect setting for confirmation bias: every team picks the slice that favors it, and the meeting turns into a chart duel.

Confirmation bias: when you know the answer before you ask

It is the most common bias and the hardest to spot in yourself. You have a hypothesis, you look at the data, and you stop looking the moment something agrees with you.

A typical case: the director of a cosmetics store believes organic Instagram is her best-selling channel. She opens GA4, filters the week a post went viral, sees 140 orders from social and takes the screenshot to the meeting. She never opened the other 11 weeks of the quarter, which averaged 18 orders a week from the same channel.

The 140 is real. The conclusion is not. The quarter had 338 Instagram orders (140 + 11 × 18), about 28 a week on average, and a big share of them came from a single event that does not repeat on demand.

How to defuse it

• Before opening any report, write down which result would change your mind. If none would, you are not analyzing.
• Set the period and the slice before you look at the numbers, not after.
• Ask someone who disagrees to build the analysis from the same data. The gaps between the two versions show where the bias is.

Survivorship bias: you only see who stayed

The classic story is about World War II planes: the military wanted to armor the most damaged areas of the planes that came back, and statistician Abraham Wald pointed out that the critical hits were on the planes that never returned. In marketing, this happens every week, just without the planes.

Here is a worked example. A fashion e-commerce brand gained 1,000 new customers in January. By June, 620 of them had not bought again. The company sends a satisfaction survey to the active base, 380 people, and gets 152 replies. The average score is very high and the team celebrates its “excellent service”.

But the survey heard from 15% of January’s customers, and precisely the most satisfied 15%. The 620 who left, who hold the most valuable information about what goes wrong, were never even asked.

Other everyday versions: studying only the ads that worked to copy their formula (without looking at the ones that used the same formula and failed), analyzing only closed deals in the CRM, or measuring retention only among customers who made it past month one.

How to defuse it

Always ask: who is missing from this table, and why? If the answer is “those who gave up”, “those who did not reply” or “the ones we paused”, the sample is filtered by the very outcome you are trying to explain. Cohort analysis helps a lot here, because it follows the whole group from day one, including the people who disappeared.

Correlation does not imply causation: what does it mean in practice?

Two metrics rising together does not mean one caused the other. Often a third, hidden factor is pushing both at the same time.

A November example: an electronics store notices that in the weeks it posted more on Instagram, it sold more. The natural conclusion is to double posting frequency. But the weeks with more posts were exactly the Black Friday weeks, when the marketing team produces more content and customers buy more anyway. The calendar moves both variables.

The same mistake shows up when someone says “customers who use coupons buy more often”. Maybe coupons build loyalty. Or maybe loyal customers are exactly the ones who receive and redeem coupons. Without a comparison group, there is no way to tell.

How to defuse it

• List everything else that changed in the same period: seasonality, price, stock, paid campaigns, site changes.
• Compare with the same period last year, or with a region that did not get the action.
• When the decision is expensive, test it: split a group that gets the action and one that does not, at the same time.

Other biases that cost money (and go unnoticed)

Regression to the mean

A campaign with an average CPA of $30 hits $48 in a bad week. The manager swaps the creative and, the next week, CPA drops back to $33. It looks like the new creative saved the campaign, but extreme results tend to drift back toward the average on their own. Anyone who always intervenes after a bad week will always think they got it right.

Small samples

Three sales from 25 visits is a 12% conversion rate. One sale fewer drops it to 8%; one more lifts it to 16%. At that volume, the difference between two pages tells you nothing. That is why diagnosing a sales drop means checking volume before checking rates.

Anchoring

This year’s target is “grow 20% over last year” because last year is the number on the table. If last year was unusual, up or down, the target inherits that distortion without anyone noticing.

A pocket checklist for your next meeting

Bias	How it shows up	The question that defuses it
Confirmation	The chosen slice favors whoever is presenting	Which result would change my mind?
Survivorship	Only those who stayed, bought or replied are counted	Who is missing from this table?
Correlation vs. causation	Two metrics rise together and one becomes the explanation of the other	What else changed in the same period?
Regression to the mean	Improvement after acting at a peak or a trough	Would this number have recovered on its own?
Small sample	Rates compared on a few dozen cases	How many cases sit behind this rate?
Anchoring	Targets set from an unusual reference number	Why this baseline and not another?

In practice, nobody removes bias through willpower. What works is ritual: the same questions, asked every time, before every relevant decision. Teams that take this seriously usually build the checklist into the agenda of their data-driven meetings, and that is where data literacy stops being training and becomes habit.

Frequently asked questions

What are the most common types of bias in data analysis?

The ones that show up most in companies are confirmation bias, survivorship bias, confusing correlation with causation, regression to the mean, conclusions drawn from small samples, and anchoring. The first three account for most bad decisions in marketing and sales.

What is survivorship bias? Is there a simple example?

It means drawing conclusions only from those who “survived” a process. Example: measuring customer satisfaction by asking only people who still buy from you. Those who left, and who would explain the problems, are missing from the sample.

What does “correlation does not imply causation” mean?

It means two variables moving together does not prove one drives the other. A third factor such as seasonality may be moving both, or the relationship may run the other way. To claim causation you need a comparison group or a controlled test.

How do you avoid confirmation bias in an analysis?

Define the question, the period and the success criterion before looking at the data, and write down which result would change your mind. Having someone who disagrees review the analysis with the same data also helps a lot.

Does AI eliminate bias in data analysis?

No, but it helps reduce it. An AI that looks at the full period and every source does not pick the slice that flatters someone. On the other hand, if the question is already biased or the input data is filtered, the answer inherits the problem.

The right question beats the pretty chart

Bias is not a lack of intelligence. It is the brain’s normal way of saving effort, which is why it hits the intern and the CEO alike. The defense is not to distrust everything but to make checking cheap: the easier it is to cross sources and look at the whole period, the less room there is for the convenient slice.

That is exactly what Sherlok does: you ask a question in plain language, it crosses Meta Ads, Google Ads, GA4 and your CRM over the full period and returns a ready-made analysis, including whatever contradicts your hypothesis. It becomes much harder to see only what you wanted to see.