Big data for predictive analysis: hype or reality?

The past 12 months have seen the information management and business intelligence world abuzz with the idea of big data. The technology behind big data promises to change the information landscape and provide a strong competitive advantage and insight previously considered unfeasible.

Having investigated the realities of big data, we’ve come to a rather unconventional conclusion. Now that the initial excitement is over it has become apparent that while big data has many practical uses, for most businesses, big data for predictive analytics is simply hype. Big data is a technology looking for a business problem to solve. Most companies looking into this technology fail to incorporate the solution into the business decision-making environment and are failing to make effective business use out of it. Starting with a clear, well-articulated business strategy around information management assets is the best way to proceed.

From a practical point of view, most organisations struggle to handle the volume of structured data they already have. Add to this the extra pressure of overlaying unstructured data, as is the case in big data, and companies are faced with far more data than they can realistically analyse. It can potentially just mean a big data headache for the information management groups.

Big data carries the danger of simply creating unnecessary and futile noise. Big datasets often contain significant information redundancy such as irrelevant tweets, useless video and junk data.

Typical organisations also struggle to find appropriate skills in the marketplace for big data technologies. They need data scientists who can articulate the business problem as a set of data questions and then manage and analyse data at scale. These are rare and expensive resources.

Perhaps most importantly, vastly improved analytical accuracy does not come from additional data as, beyond a statistically significant sample, the increases in predictive accuracy are actually insignificant.

Savvy businesses will find sophisticated analysis and strategic direction not from big data but by defining analytical solutions that focus on:

• Clear business objectives;
• Improving the definition of appropriate variables and samples;
• Improving the implementation of insights throughout the value chain.

The suggestion is not to reduce the amount of data used by analytics today but rather to use the analytics model on existing data to get effective results. True value comes not from big data but smart, fat data.

Fat data and smart data

Rather than adding more data, most organisations can benefit from identifying the most appropriate variables to their models. We call this fat data. Fat data is rich, deep and full of insightful information about a business. It enables companies to learn more, look further and use their information in more varied ways.

Becoming shrewder with the data an organisation already holds is what we call smart data. Smart data looks at making the most of available data (both structured and unstructured) and concentrating on specific signals rather than trawling vast amounts of data.

Creating data that works

In order to create fat data and smart data, an organisation should apply the same approach as they would to any strategic business issue. The key to getting the most out of data is to focus on the business problem that needs to be solved and articulating that.

A business problem can be anything from getting better traction with direct marketing to reducing risk or fraud; from identifying the most profitable accounts and why those are so profitable to targeting resourcing efforts toward selling to the most likely purchaser. To get meaningful insight from data, organisations should map out exactly what they want to change, grow or reduce and then work on how to get there. This approach is far more likely to reveal the answers and get results than big data.

Data lakes: value if used properly

While most organisations will not realise true benefits from big data analytics, there are areas of value for those companies that have the capacity and skill base to yield results. The ‘data lake’ concept, where organisations use big data technologies to democratise enterprise information, can prove successful to those at the forefront of technology.

Despite the hype, most organisations don’t need to adopt big data to realise significant benefits from the use of data analytics and statistical techniques. By improving sampling and properly utilising the information that currently exists, organisations can uncover meaningful analysis for better business decision-making. The best advice we can give is to focus on the business problem and work with existing data before considering big data.