"We're a data-driven team." It's one of the most common claims in project management, and one of the least examined.
Everyone means it sincerely. Almost nobody checks whether it's actually true of their own decisions - because the moment you looked for data to confirm something you already believed, you stopped being data-driven and started being data-decorated.
This is the skeptical companion to everything else in this blog's data analysis thread. Exploratory Data Analysis is an Art is about looking at data honestly. Your Project Dashboard Is Lying to You is about summaries that mislead by accident. This post is about something less comfortable - the ways we mislead ourselves, on purpose or not, while sincerely believing we are following the evidence.
The Claim and the Reality
"Data-driven" implies a direction: The data leads, the decision follows. In practice, the order is very often reversed. You already have a preference - keep the current vendor, defend the sprint plan you built, justify the deadline you promised a client - and then you go looking for the metric that supports it. The metric existed before the decision was made in your head. The data didn't drive anything. It was recruited afterward.
This isn't dishonesty, most of the time. It's confirmation bias: the well-documented tendency to notice, trust, and remember information that fits what we already believe, while giving contradicting information less weight or skipping over it entirely. It happens without anyone intending to deceive anyone, which is exactly why it's hard to catch.
How it Actually Happens on a Project
A few familiar patterns, none of which feel like bias from the inside:
The vendor you already liked. Two vendors are compared. One has slightly better cost numbers; the other has slightly better delivery-speed numbers. Whichever vendor the decision-maker already preferred tends to have their strong metric presented first and their weak metric mentioned briefly, if at all. Nobody decided to be unfair. The metric that confirmed the existing preference just felt more "relevant" to highlight.
The sprint that "went fine." A sprint with mixed results gets summarized using the metric that looks best - velocity held steady; while the metric that looked worse - a spike in reopened tickets; gets left out of the update because "it's probably noise." It might be noise. It might not. The decision about which explanation to believe was made before anyone looked closely at either number.
The deadline everyone wants to hit. A schedule is slipping. The forecast based on recent velocity says it won't be met. A different forecast, based on an optimistic "if nothing else goes wrong" scenario, says it might be. Guess which one makes it into the stakeholder update. This is sometimes called the planning fallacy when it happens to estimates, but the same mechanism drives which existing data gets cited to support a date everyone already wants to be true.
The dashboard metric nobody questions because it's trending up. Once a metric starts trending in a favorable direction, it tends to get cited more often and scrutinized less - not because it's more reliable, but because favorable numbers invite less scrutiny than unfavorable ones. A metric going the "right" way quietly earns a pass that an unfavorable one never gets.
The Two Failure Modes Hiding Inside "Data-driven"
Cherry-picking is the most direct version: selecting the data points that support a conclusion while leaving out the ones that complicate it. It's rarely a conscious act of deception. It's much more often the simple result of stopping the search for information the moment you find something that agrees with you.
Goodhart's Law is the subtler, more structural version: once a measure becomes a target, it stops being a reliable measure, because people - often without realizing it - start optimizing for the number rather than the thing the number was supposed to represent. A team measured on velocity learns, consciously or not, to estimate in ways that make velocity look good. A team measured on tickets closed learns to close easy tickets first. The dashboard still moves. The underlying reality it was meant to track quietly decouples from it.
Goodhart's Law: https://en.wikipedia.org/wiki/Goodhart%27s_law )
Both failure modes produce the same outcome: A decision that looks evidence-based because there's a chart behind it, while the chart itself was selected or shaped by the decision rather than the other way around.
A Short Checklist for Catching Yourself Doing It
None of this is solved by trying harder to be objective - that rarely works, because confirmation bias doesn't feel like bias while it's happening. It's solved by building a few specific checks into how you look at data before a decision, not after one.
- Write down your answer before you look at the data. If you already know what you expect the numbers to show, you'll notice immediately when you start searching harder for confirming evidence than disconfirming evidence.
- Ask what data would change your mind. If you can't name a specific number or finding that would make you reconsider, you're not really testing your position - you're building a case for it.
- Actively look for the metric that argues against your preferred answer. Not generically "be open-minded” - specifically go find the one number most likely to complicate the story, and look at it before you finalize anything.
- Notice which numbers you stopped questioning. If a metric has been trending favorably for a while and nobody's dug into why, that's worth a second look - not because it's necessarily wrong, but because favorable numbers get a free pass by default.
- Ask who chose which metrics made it into the report. If the answer is "whoever wanted a particular outcome," that's worth knowing before you trust the report.
- Get someone who disagrees with you to look at the same data. If their read is very different from yours, that gap is informative - not something to resolve by finding one more chart that backs you up.
The Conclusion
Being genuinely data-driven isn't about having charts behind every decision, it's about letting the data have a real chance to change your mind before you've committed to an answer.
The uncomfortable test isn’t - do we have data to support this? Almost any decision can clear that bar. The real test is - did we go looking for the data that could have talked us out of it, and did we actually look at what we found?
Most status reports never ask that question. The next one you build, try asking it before you finalize the metrics that go in it.
No Worries
