Want to tap generative AI? Get a handle on your data

Nobody in business right now could have missed the rapid rise of generative artificial intelligence (GenAI) and the novel mix of use cases for which it’s already being deployed. But none of this would have been possible without the careful management and manipulation of data.

Businesses have latched onto the GenAI wave quickly and are using it for everything from writing marketing content to voicing chatbots. NAB is using AI to not only help fight fraud, but also to help employees prepare credit reports and even draft emails to customers. Atlassian, meanwhile, is tapping AI to create new customer experiences and ways of working.

None of these activities are new, of course, they’re just becoming much, much easier now as generative AI steps in to tackle the time-consuming job of synthesising large amounts of information and turning it into something usable. Although some of these use cases are in their infancy, that doesn’t mean they won’t become widespread applications for GenAI.

Digging up the data

As useful as it is, GenAI technology based on large language models (LLMs) like that used by ChatGPT requires enormous amounts of data to churn out anything even remotely useful. For businesses looking to replicate the usefulness of such AI programs internally, the biggest stumbling block usually lies in the data feeding the algorithm.

It’s one thing to turn to a platform like ChatGPT to create a travel itinerary based on publicly available data drawn from the internet, but it’s a very different proposition to train a customer service chatbot to answer specific questions about proprietary products with commercially sensitive data that cannot be made publicly available.

That is why, when it comes to enterprise GenAI applications, businesses need to do much of the data management work themselves if they want to protect their proprietary assets and truly trust the data underpinning the quality of the AI models being used. This gives rise to cost and complexity challenges as well as data and security governance considerations.

But it doesn’t have to be that hard. As new AI applications emerge, new data platforms are emerging to tackle the evolving challenges for enterprises wanting to leverage AI. At Databricks, we refer to this new generation of unified systems as “data intelligence platforms”, and they have the ability to deeply understand an organisation’s data while keeping it safe and secure.

New platforms for a new era in AI

Data intelligence platforms represent a step change in the ongoing evolution of data management, one that enables businesses to democratise access to data, automating manual administration and enabling turnkey creation of custom AI applications. Built on top of a data lakehouse architecture, a data intelligence platform provides companies with unified security, governance, data storage, management and secure sharing.

As its name suggests, the lakehouse model combines data warehouse functionality with data lake capabilities to create a central location that can hold large amounts of data in its native, raw format, while still ensuring it is easy to locate and retrieve. This provides an open, unified foundation for all data and governance in the organisation.

What brings the ‘intelligence’ factor to next-generation data intelligence platforms is the use of AI models to deeply understand the semantics of enterprise data. These AI models build on the foundations of the data lakehouse and enable natural language access, semantic cataloguing, enhanced governance and more. In essence, AI-powered intelligence enables businesses to make more sense of data across the enterprise.

AI leadership across industry

The need to make more sense of data is something happening across industry verticals, especially as the widespread use of AI takes off. The financial services industry is experimenting with GenAI, as is the mining industry. But we’re seeing adoption across the board. Where there’s a valid use case, organisations are tapping into it, no matter what industry they’re in.

Just as Australia’s public cloud adoption was among the highest in the world, we’re seeing this early adopter energy shift to focus on GenAI — indeed, all kinds of AI.

Given the volumes of data needed to drive GenAI and the benefits it can offer, the starting point has to be the data platform. Once we get this part right, Australia can take its rightful place as a global leader in the area of GenAI — just as it did with public cloud.

Image credit: iStock.com/DamienGeso