Five key data trends Australian IT leaders need to know about this year

Data can be the greatest advantage to a business, providing valuable insights for informed decision-making and innovation, both of which are critical for organisations navigating the current macro-economic environment. However, the changing nature and increasing complexity of data can overwhelm businesses even as they try to make it work for them.

For instance, a recent study by Dell Technologies reveals only 11% of ANZ IT decision-makers say all their innovation efforts are based on data, citing technology complexity, cybersecurity threats and lack of IT infrastructure to process data at the edge as key barriers. In other words, 89% of businesses are missing opportunities to improve outcomes.

If a business is among the 89%, the first step to overcoming obstacles and taking back those missed opportunities is awareness. To untangle the complexity and understand the current data landscape, we need to be aware of five key data shifts.

1. The exponential growth of structured and unstructured data

Fuelled by an abundance of smart devices and IoT sensors, worldwide data creation has been soaring for more than a decade. A study from IDC notes that between 2021 and 2025, new data creation will grow at a compound annual growth rate (CAGR) of 23%, resulting in approximately 175 zettabytes (ZB) of data creation by 2025. To put this into context, a single zettabyte is equivalent to a trillion gigabytes.

In addition to that, more data forms — including unstructured and streaming data types like loose files, PDFs, photos, audio and video clips — are growing at an unprecedented rate. The same IDC report states that 80% of worldwide data will be unstructured by 2025.

Finding a solution for managing unstructured data is a challenge. Next-generation applications that could potentially manage rapidly growing unstructured data typically require the extreme performance of all-flash storage, but budget pressures have made it hard for organisations to commit to the new applications that these powerful systems require.

Businesses will need powerful scale-out file storage solutions that are easy to scale and use, no matter how much unstructured data the environment needs to manage.

2. Emerging technologies are sparking a new era of intelligence at scale

Edge computing, 5G, AI and machine learning (ML) are transforming how data is being collected, processed and used. For the first time in history, we’re meeting the explosion of data with intelligent infrastructure, software and algorithms to rapidly turn it into actionable information.

This data can be used to create new value and drive better user experiences at the edge, with a symbiotic relationship between the advanced, connected technologies being deployed to thrive in the digital economy and the wealth of new data waiting to be uncovered. Likewise, edge technologies and data management will be much more successful if they can work together.

The rise of AI is also creating new challenges and opportunities, as organisations come to grips with how this transformative technology will change their operations, and as they roll out internal AI programs designed to leverage corporate data and put AI-generated insights at employees’ fingertips. AI has the potential to create massive efficiencies for companies, and to relieve staff of the burden of repetitive, mundane work in favour of tasks requiring creativity and strategic thinking. Every organisation should be thinking about how to use AI to become more agile and efficient and to better serve customers. To do that, however, the data needs to be ready.

3. Decentralised data

The adoption of emerging technologies has led to more distributed locations where data originates. As data’s focus rapidly moves toward the edge, data is increasingly being stored, processed and acted on closer to its source.

But, as more functions take place at the edge, businesses will need to manage data differently and consistently — from the core to across edge and hybrid clouds. That requires changes to computer, network, storage and application architectures.

4. Rising consumer expectations

Today’s consumers are more empowered and are demanding more data-rich, personalised and real-time experiences. A 2023 study by Cheetah Digital found consumers crave personalisation as much as they crave privacy, with 60% saying personalised offers after staying on a brand’s site for two minutes are ‘cool’ and 65% saying a reminder or an advert about an abandoned shopping cart is also ‘cool’.

In the past, organisations could take days to come up with new data insights, but in today’s world that’s far too long. The increasing reliance on AI and ML to make real-time decisions in a distributed environment can strain even the most advanced data management strategies.

Most organisations currently lack the IT capabilities to keep up as their data management systems are severely outdated and cannot keep up with today’s standards. To match consumers’ expectations, organisations must advance in-house data processing capabilities and ensure that systems and processes are efficient and proactive.

5. Data breaches versus the regulatory environment

While the second half of 2022 saw a 26% increase in data breaches and increasingly sophisticated cybersecurity threats, including from AI, the most recent figures from 2023 indicate the situation in Australia has eased somewhat, with statistics from the Office of the Australian Information Commissioner (OAIC) finding the number of data breaches in the six months to June of this year was down 16% compared to the previous reporting period.

Despite this trend, businesses and consumers can’t ease their vigilance about good cybersecurity practices. Cybercriminals constantly evolve their tactics, finding new weaknesses and vulnerabilities, and the rise of generative AI has made it easier for crooks to create believable-looking phishing emails, fooling people into handing over their credentials and allowing bad actors easy access to logins, corporate networks and accounts.

The regulatory environment in Australia is also evolving to deal with the changing threat landscape, with the Australian Government recently increasing the maximum penalty payable by corporations that suffer a data breach and lose customer information to $50 million.

As more data is collected, stored and processed in multiple locations, the attack surface for malicious activity also grows, making compliance with Australian data laws and regulations more complex. In addition, customers want to do business with organisations they can trust with their data.

These trends underscore the ways data users and consumers have changed, and how organisations will need to adapt to stay relevant.

Simply being digital is no longer a differentiator

With zettabytes of data freely available at our fingertips, businesses must look inwards and ensure that in-house processes can act and react to the data they’ve gathered. Only then will they have access to the intelligent insights you need for the business to innovate and succeed in this data-driven world.

Image credit: iStock.com/Maxger