Getting started with big data

Organisations in the Asia-Pacific region are growing increasingly keen on big data initiatives. Where 2013 saw growing big data interest from commercial and academic sectors, 2014 has seen increasing interest from government organisations in the region.

That’s according to a new report from analyst firm Gartner, called ‘Big Data: First Aid Tips for Creating a Structured Approach for Organizations in Asia/Pacific’, penned by the firm’s Daniel Yuen, Ian A Bertram and Craig Lawson.

Big data “continues to be an important topic requiring much discussion within organisations - trying to decide where to start, discovering the best practices to bring big data into the organisation effectively, and determining what the right architecture is that organisations should embrace”, the authors write.

But there are several problems that organisations commonly run into.

Where to start

Firstly, Gartner says, some struggle to work out where to start with a big data initiative. Some make the mistake of taking a technology-first approach. Picking a technology and running with it could lead you to (unwisely) approach every big-data-related issue with that one technology. In reality, the best approach with big data often involves a mixture of many solutions, the analysts say.

“The first questions that should be answered specifically are what business problems or business opportunities you are trying to address in implementing big data; and what outcome you are trying to achieve - not simply asking, for example, what type of Hadoop engine you should buy,” the authors write.

Starting with the business objective means that the other components - like what data sources you need, the use case and its business impact, and how to measure the project’s success - will logically follow. These business objectives fall into four main categories:

• Operations matters like efficiency, cost reduction
• Customer-related matters like customer experience or customer service
• New business, including new business models, products or new markets to break into
• Risk-management concerns, like fraud detection and compliance

The analysts also advise that these initiatives should start small, as a proof of concept. “The organisation will develop the skills and capabilities as this initiative matures.”

A structured approach

Organisations also find that creating a business plan to justify investment in big data initiatives can be difficult. In this case, the analysts suggest thinking about big data business opportunities, which typically fall into three main categories:

1. Making better-informed decisions. “By embracing a broad range of data sources, organisations can better understand the world around a particular business problem”, helping managers and executives make better decisions, the analysts say.
2. Discovering hidden insights. “Some insights are not obvious without probing into large sets of detailed data.” Drawing on large pools of data - for example, data going back many years - allows organisations to come to conclusions they otherwise would not have been able to.
3. Automating business processes. Big data can, in some circumstances, allow organisations to automate businesses processes that were previously too expensive to implement, the analysts say. “For example, an insurance company automates the fraud detection on all claims submissions. This is based on text analytics technology to identify any possible fraud, analysing historical records, claims types and claims patterns, and comparing it across recent submissions.”

The business opportunity you seek will determine which technologies, approaches and skill sets your project requires, the analysts say. The most commonly used components are:

• Distributed batch processing for data at rest
• Complex-event processing
• Distributed stream computing platform for real-time analytics and monitoring
• In-memory databases
• Data grids to enable interactive data analysis
• Graph databases to represent data that is impossible to translate into traditional data representations

“Starting with your business objectives or risk-reduction requirements, line up the business opportunities sought and map it to the components required. Create a clear line of sight from the business objectives to the technologies investment. This provides a solid foundation for the justification on the business return of technologies investment,” the analysts write.