Diversity the key to managing big data

Big data - the accumulation and subsequent mining of large data stores - carries great potential for businesses looking to gain insight into their markets or customers. But while the concept of big data is simple, the management of big data is not.

Big data: it’s the industry buzzword describing the phenomenon of businesses accumulating, storing and mining large files (400 GB to terabyte) in large volumes. The big data moment has been years in the making, as information reaches new mass, variety and richness.

The proliferation of large-scale data sets has been brought about by businesses operating in rich data formats, such as using audio or video files, or by their willingness to accumulate enough information to develop statistically significant insights into their market, such as saving posts on social media networks.

For business, data is essential to both strategy as well as business continuity. Put simply, content is knowledge, so content is king.

The promise of big data is not unfounded. The generation of insight and the identification of new opportunities from the collection and summation of data can readily support a company’s growth strategy, by providing concrete, analytical basis on which to make decisions. However, it is imperative that big data sets do not become an avalanche of unstructured data which prove to be a huge time and cost impost. The importance of adequate IT architecture, namely suitable storage solutions, cannot be understated. Without the right kind of storage, the sheer volume of data cannot be easily accessed nor readily deployed.

Given that the ‘big data’ phenomenon has been years in the making, the question CIOs should be asking themselves at this stage is how can I balance a range of technologies to support a strategy to store and protect my data? Finding the right tools for data management is integral to effective data protection; that is, the security of knowing that data will remain in existence and be valuable when required.

The question of what constitutes an effective storage system finds its answer in diversification. Diversification is the antithesis of the ‘one size fits all’ approach to big data management. By matching data to a suitable storage system, by taking into consideration when and how it will be used, organisations can ensure that storage solutions support, not hinder, key business drivers such as efficiency and continuity. Through this, businesses can position themselves to navigate this new era of extensive and expansive information with confidence.

Data storage diversification: backup versus archival

The key to managing big data is setting strategies to archive data in an automated, reliable and cost-effective way. The big data phenomenon means that organisations are faced with the challenge of contending not only with a large volume of data but also a wide variety of data formats. What is emerging as an effective approach across all industries is diversification - a model of data storage which involves a range of products to support an information management strategy. These products can include automation, disk and deduplication, software and backup and archival. The principle underpinning this approach is the insistence on the right storage media for a particular kind of data. Businesses need to recognise that there is no silver bullet when it comes to data storage and protection; in reality, a suite of solutions aligned to various functions is required.

Big data management requires a range of technologies

One particular aspect on which CIOs should focus is the methodology of backup and archival, as this is one of the more obvious ways in which different kinds of files can be distinguished in a business environment. When businesses need quick, regular access to data, primary, disk-based storage is the most suitable. This is the kind of data which can be regularly backed up to ensure its availability. Contrastingly, as data gets older and is accessed less often, organisations can reap significant cost advantages by moving older data to lower-end disk or tape, freeing up costly primary storage.

By moving older data to these types of media, companies reduce the amount of disk required. Archiving is a key aspect of a holistic and cost-effective data storage solution. This diversified model is useful for organisations needing high performance and lowest, long-term storage cost. In separating storage format according to data usage, businesses can optimise their operational workflow. In doing so, they can better navigate big data files, easily streaming media content or manipulating large analytical data files, because they are stored in a medium most suited to their format and usage patterns.

Diversification in a data storage solution is essential if businesses wish to transform their IT infrastructure into something that provides value for business objectives, rather than simply being a cost centre that slows people and processes down. A considered mix of technologies, with the core methodology of back-up and archival, saves IT resources, reduces strain on IT personnel and can scale with a company’s needs.

Ultimately, the prevalence of new types of digital data, the sheer volume of data and the imperative to meet ever more burdensome compliance requirements means that now is the time for business to put in place the right technologies, supporting the right strategy, to manage and protect their most valuable asset: their information.

By Bruce Coulthard, Senior Solutions Consultant A/NZ at Quantum, where he specialises in backup, recovery and archive solutions architecture, consultation and education. He has more than 10 years of experience in the architecture, design and implementation of medium- to large-scale storage and data protection solutions and is well versed in aligning data protection solutions and strategies with technical and business needs across the industry.