The human flaws influencing big data

Will big data - the collecting of vast amounts of data about nearly everything - finally help us predict the future? Organisations spend billions of dollars annually on forecasts to reduce uncertainty about the future including stock analysis and performance, weather forecasting, technology forecasting and market research.

Data is portrayed in popular media as able to assist and improve decision-making of all types, including crime reduction, product development and investments. Television shows like Numb3rs and Person of Interest portray the use of mathematics and algorithms to predict behaviour. Nate Silver, an American statistician and writer, recently ‘predicted’ the outcome of the US Presidential election, correctly calling the outcome of 49 states.

Big data, the potential to have all the information necessary to make decisions, offers the promise of improving decisions and business performance. However, data is one input to the process - what about the impact of humans? There are three points in the decision process susceptible to the behaviour of humans: choosing which data and how it is collected; analysing the data including both what data is used and how it is weighted; and interpreting the data and making a decision.

It has been said greater amounts of data often does not lead to better decisions. Many people believe more data can decrease the quality of a decision. With large amounts of data, it is easier to believe correlation is actually causation. But just because things may happen in what seems to be a pattern, one action is not necessarily causing the other. Greater amounts of data can lead us to falsely conclude that the data ‘proves’ what it is we want to believe.

Greater quantities of data can lead to too much information being collected, which can increase complexity and lead to overanalysis and reliance on the data. Humans must still be relied on to manage these factors around the decision process, and humans are subject to cognitive biases, which influences the way we process and interpret information.

When having vast amounts of data available for analysis and decision-making, there are several types of cognitive bias that are particularly relevant: the gambler’s fallacy, framing bias and the ambiguity effect. The gambler’s fallacy is giving greater weight to previous events, believing they impact future outcomes (the best example is flipping a coin). Secondly, every set of data used for decision-making can be presented to reflect a certain point of view, or framed. Think of a debate: both sides use the same information in support of opposing views.

Thirdly, when people are given a choice between two possibilities - one with a known probability and one with an unknown probability - people have a tendency to choose the first option. This is called the ambiguity effect and can lead to decision-makers giving more weight to some data.

The availability of data has the potential to improve decision-making. However, humans are still needed to make decisions and the better we can understand the human bias which influence the decision-making process, the better use your organisation can make of its data.

Image credit ©stock.xchng/profile/vjeran2001