Big data: it’s all about smart people

Big data is not about the technology. Big data is not about the next big thing in the IT department. Big data is not all about the social media revolution. Big data is all about really smart people forging the tools to enable business and organisations to stop guessing about which decision is the right one to make.

As society has become more digitised and storage has become cheaper and more available, more of our digitised life is within reach.

This digital asset contains enormous insight into our world, but most organisations have failed to tap the surface of this value.  Many organisations have struggled to tame their own data assets (forget social media - I am referring to things like customer communications, what people have bought, what they did on the company web page) into beautifully crafted data schemas stored in a data warehouse, that either never made it into production or didn’t contain what the business needed when it did. This has been terribly distracting, shifting the focus away from bridging the chasm between an organisation’s data assets and its business.

Big data is not a technology but a paradigm.  Store everything forever and extract value with science. Open source technologies, like Hadoop, developed out of desperation by the world’s largest internet properties, provided the world with a cheap and scalable way for organisations to store everything forever. Decades of developments in the science of machine learning, an area of research in which NICTA is a recognised international leader, has provided us with the tools to extract value. 

This brings us to the new hot career of a data scientist. A data scientist is typically not a person but a tightly integrated team of people that have a deep understanding of machine learning and can implement algorithms in software at scale. Successful data science teams, such as the Ambiata team at NICTA, have a passion for understanding the organisation and solving problems, and the creativity and lateral thinking to forge data into something the business can understand and use. A data science team can turn data into granular actionable insights for an organisation, For example, how you stop the next customer from leaving, where you send inventory, which power transformer is about to fail or whether you can waive a requirement to charge mortgage insurance. 

The challenge organisations are going to face is that if they want to become truly data driven, it is not about vendor decisions or hardware but gaining access to an experienced data science team. 

Will we see software tools replace data science teams? In my opinion: unlikely. For the same reasons that accounting software has not replaced chief financial officers. Access to a good data science team will soon become an organisation’s strategic advantage.

Once the hype dissipates, businesses will increasingly understand that big data is all about having the smartest people in the room.