Guest Columnist
As our ability to collect and analyze massive swaths of data continues to grow, the “big data” movement represents a paradigm shift in the way that businesses, governments, hospitals and other organizations make decisions. Each year, we become better and better at predicting important outcomes such as employment trends, criminal activity and medical diagnoses.
The power and potential of big data make it seem like a magic bullet for sound, unbiased and effective decision-making.
But many discussions of big data fail to address the users of it: humans. Big data is simply a tool that provides information; it’s still up to a person to integrate the information into a decision.
Often overshadowed by the technical advances in mathematics and computer science, research in behavioral science literature reveals a dark side of big data.
For example, due to a strong confirmation bias, people are likely to use data merely to reinforce their own preconceived opinions. Don’t like the output of a statistical analysis? You may be tempted, even on a subconscious level, to tweak those “flawed” parameters and recalculate before submitting that report to
the boss.
Humans also have a tendency to be highly overconfident in their abilities, which creates a problem when they are “behind the wheel” of high-powered, complex big data tools. Do you really understand the assumptions driving the calculations in that analytics software? Just go with the flow; you know what you’re doing, right?
Beyond the cognitive biases that may sway people to misuse data analytics, other behavioral research suggests that some may eschew it altogether.
Several studies have found that professionals (from managers and physicians to university admissions committees and parole boards) are often reluctant to use decision support tools like big data.
Put yourself in their shoes: You went through all that education, training and years of hard work to become a skilled decision-maker in your field. Are you really going to think it’s fair or acceptable to relegate your decisions to an algorithm or computer software, especially when it disagrees with your personal assessment?
Likely not. And some research suggests that your customers or clients may even view you negatively if you let technology make decisions for you.
For example, a recent study in the journal Medical Decision Making involved asking people how they felt about doctors who use computer-assisted diagnostic support versus those who do not. Results indicated that, on average, ratings of professionalism and patient satisfaction were higher in the latter group. Regardless of efficacy, many people don’t feel comfortable letting technology steer their decisions. Self-driving cars, anyone?
So what is the solution? Should we invest in education about the virtues of big data to help push aside our “error-prone” gut feelings or intuition, and grasp the power of rational analysis?
Unfortunately, the answer is not quite that simple.
An emerging body of research on intuition suggests that rational analysis shouldn’t be the only seat at the big data table.
For instance, some studies suggest that intuition can actually be more effective than rational analysis for certain judgments, such as those involving creativity or morality.
Intuition is also helpful when a quick decision is needed, especially when people have acquired domain expertise. Examples include firefighters making rapid assessments of how to extinguish a structural fire or neonatal nurses making snap judgments about the medical needs of a high-risk infant after birth. Intuition can shine in cases like these.
Additional insights about the power of intuition are highlighted in a recent study I conducted with colleagues at Michigan State University and the University of Kansas. We studied U.S. Air Force captains to examine the impact of leader intuition on team performance. Our results indicated that in situations requiring a significant amount of information processing, teams achieved higher performance levels when leaders used intuition in decision-making, relative to those who did not.
Why? Our results suggest that intuition helped these captains to develop a holistic, big-picture understanding of the complexities of their situation, enabling them to facilitate better team coordination and performance.
Combining the strength of intuition with the insights of big data may be the next frontier for effective decision-making, and ongoing research is exploring this possibility.
The bottom line: Despite all the hype, we shouldn’t merely outsource decision-making to big data and let the numbers do all the talking. We need to find ways of reducing the influence of cognitive biases that can lead data analytics astray, while at the same time leveraging the power of human intuition.
Dustin J. Sleesman is an assistant professor of management at the Alfred Lerner College of Business and Economics at the University of Delaware. His teaching and research focus on decision-making in organizations, group dynamics, and negotiation.
