Data-driven decision making frequently requires experimentation as a mechanism to acquire the necessary data. When working with companies in transitioning from opinion-based to data-driven decision making, however, I frequently run into push back. The typical comment is that we already know the answers to the specific questions at hand and since we already know this with sufficient certainty, the cost associated with running an experiment is just waste.
The idea that we already know, of course stands in stark contrast with reality. As I have shared in earlier articles (such as here), our research shows that half or more of all the features in typical software systems are waste as due to lack or total absence of use. Still these features were prioritized by individuals, teams and companies that knew that these features were needed. When the accuracy of your decision making processes is less than the flip of a coin, I can’t help wonder whether we should explore other ways of working
When reflecting on why I keep running into situations where people claim that they already know, I think that there are at least three aspects of human psychology that are at play here. First, there is the notion of “losing face or reputation”. In most companies, there are individuals that have been promoting certain functionality and features for a long time and who have built the most elaborate justifications. If the data would suddenly show that their beliefs about the customers and their priorities is wrong, this would result in a significant loss of face. In these situations, the risk of losing face is often considered to be more important than knowing the facts.
The second aspect is confirmation bias. As humans, we enter every situation with a set of beliefs and preconceptions. This bias causes us to handle new information differently based on whether the information confirms of violates our bias. This leads to a strong tendency to ignore data that does not match our bias and, where that is not feasible, to reinterpret data to make it fit our beliefs.
The third aspect is group-think. Most companies initially become successful because they know something that at the time the company was founded as controversial. This often leads to an “us versus them”, “we know better” culture in the organization. We refer to those as organizational shadow beliefs — beliefs that are broadly held across the company and that might have been true in the past, but that no longer are true. The challenge is that holding those beliefs may be part of the group identity, so questioning these is very risky as it may lead to being ostracized by the rest of the team
The only way to escape this challenge is to accept that most of the things we think we know are in fact wrong or at least unconfirmed. By questioning our beliefs and formulating hypotheses that allow us to learn whether certain beliefs are grounded in reality or not, we can ensure a much higher accuracy. The tool to test hypotheses of any type is, of course, experimentation. We can run pure statistically validated experiments, but also quasi-experiments, cross-over experiments or other techniques can provide accurate and relevant insights. The challenge is to accept that most experiments do not generate the expected outcomes in terms of business benefits, but should still be considered successful as these provide more information about the context in which we operate.
Concluding, the main point that I am trying to make in this post is that all of us walk around with all kinds of beliefs that we hold to be true, but a great many of those actually are not true at all. Although that may be fine in some aspects of our lives, it is a recipe for disaster in business. We have to stay grounded in the actual, factual reality in which we operate. We do this by maintaining a healthy skepsis concerning our own beliefs and those of our colleagues. Whenever things are even remotely questionable, the main mechanism that we have for finding out more is experimentation. Let’s try things out and accept that most experiments do not lead to immediate business benefits, but that each experiment is an opportunity to learn more about the reality in which we operate as a business. Remember: You think you know, but you’re most likely wrong!
To get more insights earlier, sign up for my newsletter at jan@janbosch.com or follow me on janbosch.com/blog, LinkedIn (linkedin.com/in/janbosch) or Twitter (@JanBosch).
