Storage system provides Gippsland Water certainty

Central Gippsland Water Region Corporation (Gippsland Water) provides water, wastewater and waste recovery services to domestic and commercial clients across central Gippsland in Victoria. The organisation distributes about 64,000 megalitres of water to 41 towns spread across more than 5000 square kilometres. Its clients also include businesses in the manufacturing, power generation, health service, timber, paper and dairy industries.

Gippsland Water manages about 100 water storage basins throughout the region and operates a resource recovery plant and agribusiness near the town of Sale.

In 2008, with a four- to five-year-old storage system supporting an ageing IT infrastructure and a limited disaster recovery capability, Gippsland Water decided to undertake a comprehensive upgrade.

“We were already experiencing limitations, as our existing storage system was incompatible with the enterprise-class VMware data centre virtualisation we planned to install,” said Ray Baillie, IM/IT and SCADA Manager, Gippsland Water. “Historically, our outsourcing provider had implemented VMware Server as a temporary measure to overcome the resource restrictions of older servers used in our demilitarised zone to support external-facing websites, portals and Citrix application publishing systems.”

The organisation also wanted to replace a subscription tape-based disaster recovery system that could see 96 hours’ worth of data lost and full recovery take five to seven days to achieve. “In the event of a catastrophic outage, we would have had to drive the latest tapes to Melbourne for our service provider to reconstitute our information and bring our systems back up and running,” said Baillie.

Gippsland Water released a tender to the market for a new storage system and disaster recovery environment. “We wanted a system that was scalable, functional and simple, and one that could establish a foundation for our organisation for the next decade,” said Baillie.

With a restricted budget, the cost of a physical disaster recovery architecture incorporating real-time replication of critical business information to a secondary site was prohibitive. However, Gippsland Water had a minimum objective of returning to full operations within 24 hours of an outage.

It also specified that the new storage system must support VMware’s virtual infrastructure software, including the site recovery manager (SRM) tool that automates system recovery in the event of a problem.

“We received 10 responses proposing solutions ranging from a hosted service to data replication,” said Baillie. “We decided the hosting model was inappropriate as bandwidth was not available at a reasonable cost and support was not necessarily available as and when required.”

Following an extensive research and review process in the third quarter of 2008, Gippsland Water selected Thomas Duryea Consulting to implement dual EMC CX4-120 networked storage systems, with one used for production and one for disaster recovery.

These systems are linked by a 100 megabits per second, full duplex ethernet connection and operate on a constant like-for-like basis. They met the organisation’s business requirements of supporting a VMware implementation including SRM, with continuous data protection (CDP) and continuous remote replication (CRR) provided by EMC RecoverPoint.

Gippsland Water is using EMC RecoverPoint CDP to deliver local ‘journaling’ in its production data centre that enables initial on-site recovery of application servers without initiating failover or disaster recovery processes.

“EMC RecoverPoint CDP enables us to roll back the virtual servers running our core applications to direct-from-disk ‘snapshots’ taken at intervals of seconds within the previous 24 hours,” said Andrew Ross, IT Team Leader, Gippsland Water. “We are also using EMC RecoverPoint CRR to fail servers over and bring them up at the disaster recovery site.”

The organisation is using Replication Manager for VMware to deliver a central management console to coordinate the process of replicating information between the production and disaster recovery data centres.

The asynchronous replication delivered by the disaster recovery system ensures at most only 15 minutes’ worth of data is lost in the event of a severe outage. In addition, full recovery of production systems can now be guaranteed within 24 hours, up to 700% more quickly than the five to seven days previously required. “I feel we are in a very good position at the moment,” said Baillie. “If we do experience a major outage, we will not go out of business.”

The virtualised environment is enabling Gippsland Water to reduce its hardware, power and cooling costs, with physical server numbers expected to fall from 30 to 10 over the next 18 months.