Harmonise amorphic data and tame the deluge

Armed with the right tools, it’s possible to extract the signals from the IT noise.

According to experts, 80 to 90% of available data is in amorphic or unstructured form. Textual data includes social chatter, electronic documents, word-processed content, blogs, presentations, emails and other record-keeping content. Non-textual data — such as images files, video, audio and analog communications — usually exists in binary format.

Amorphic data processing requires special techniques such as natural language processing (NLP), part-of-speech tagging, image processing and data mining. With open-source, big data processing software that can run on commodity hardware and democratisation of analytics, enterprises are increasingly looking to generate insights from unstructured data for revenue generation or optimisation.

Social networking providers are using online communication channels and networks for targeted marketing campaigns. By continually mining information from conversations using an ensemble of NLP models and creating network graphs of our social connects, we can segment customers based on a range of factors. However, in addition to having good demographics data and customer segments, they also use social communication channels to gauge the general sentiment and overall impact of the campaign.

Voice and data operators are using call records to measure first call resolution (FCR), which can be leveraged to analyse customer satisfaction, agent effectiveness and workforce optimisation. Interactive voice response (IVR) data is usually present in large XML files which need to be processed and mined. Voice records can be parsed using NLP and used to determine the sentiments of the caller (customer). Unifying all of these channels can help manage customer expectations better and improve their satisfaction score.

E-commerce sites use predictive models on historical content for recommendations. As a customer browses or buys products from their sites, enterprises use content-based and collaborative-based models to better predict similar items and thus improve sales and enhance customer experience. The underlying data includes a trove of customer information.

Airlines are taking customer engagement beyond mere travel to provide a more complete experience. Using information from transactional systems along with social data, personalised information regarding events, accommodation, food and dining packages are being pushed to users’ smart devices. This not only keeps the customer engaged with their preferred airlines, it also ensures loyalty in the long run, bringing in additional revenue by way of channel partners.

Banks and financial institutions can apply deep learning concepts to bolster security with face recognition software and leverage video processing for event correlation. Biometric data can be mined and used to create new business models for third-party validations for banks. These techniques can be employed for purposes ranging from personal security and workforce monitoring to national security.

Businesses can benefit tremendously by harmonising the myriad structured and unstructured data in their purview by garnering the right ecosystem of people, processes and platforms. Armed with the right tools, it is possible to tame the data deluge and extract the signals from the noise.

Ganesh Moorthy is an Apprentice Leader at Mu Sigma. His role encompasses the responsibilities of a program manager and solution architect for the organisation’s R&D engagements. He has over 16 years’ experience in leading enterprise solution development for global Fortune 500 clients and is currently involved in building industrial internet, augmented reality and analytics and visualisation platforms for both descriptive and predictive analytics.