Outback council introduces multivendor disaster recovery program

Wentworth Shire Council has deployed a more resilient IT disaster recovery program in anticipation of catastrophic natural disasters.

The shire itself is located more than 1000 km from Sydney and more than 600 km from Melbourne, and is known to locals as ‘the gateway to the outback’. The 26,000 km² region is home to 8000 residents.

The council provides an extensive range of infrastructure and services for residents and businesses, and manages land and business development, tourism and cultural and recreational services.

In early 2010, Nigel Keath, Wentworth’s Manager of Information Systems, determined that it was time to put in place a more resilient IT disaster recovery capability. Natural disasters in Victoria and Queensland had heightened awareness of the problems associated with bushfires and floods. Should anything similar ever happen in Wentworth, Keath knew it could take anywhere between four to six weeks for the council to recover its information systems using existing back-up facilities.

“Wentworth is located on the Murray and the Darling Rivers and we have had floods in the past. Also, our buildings are quite old. We’re not in a modern office and that makes the threat of fire of greater concern,” he said.

In addition, the council had for some time been gradually virtualising its infrastructure. As natural hardware replacement cycles occurred, new virtualised servers were being introduced and it was time to take the next step in this project.

The council prepared a brief seeking assistance with the implementation of two storage area network (SAN) devices - one to be used as a production site storage array and the other to form the basis of a new disaster recovery capability located in a separate council building. The brief included replication between the disaster recovery site and the production site, an update to the council’s overall VMware infrastructure plus configuration of VMWare’s Site Recovery Manager.

Above all, any proposed solution had to offer consolidation and rationalisation of server and storage hardware; improve the council’s capability to recover from an IT disaster; and be capable of improving the performance and reliability of Wentworth’s key systems.

Tecala responded with a plan involving implementation of an EMC unified storage area network, the addition of a Dell Server for the disaster recover site, the recommissioning of some existing hardware and streamlining of the current back-up and disaster recovery replication procedure.

The disaster recovery system has given Wentworth Shire Council new confidence in its systems. Through the use of an EMC Celera Replicator and a fibre connection, the council’s production storage array is replicated at the disaster recovery site every 10 minutes. Adding further resilience, VMware Site Recovery Manager provides an automated fail-over service between the two sites.

“Through these technologies, we’ve pretty much achieved synchronous replication. Ten minutes’ worth of data is all we would potentially lose in a disaster. The whole thing has been designed so that the disaster site can be packed up and rolled out the door within 15 to 20 minutes,” Keath said.

Server infrastructure has been consolidated even as demand for services has continued to grow. Storage capacity has increased dramatically while server and storage manageability have also improved. “We used to have to wait for an outage to be experienced before being made aware of any problems with the storage system. Now we automatically receive alerts so hopefully we can resolve any issues before they impact the business,” Keath said.

The amount of time required to carry out menial storage and server tasks has been reduced and deployment of additional servers can be achieved within approximately half an hour - far less than in the past.

“We’ve achieved everything we wanted to achieve. It’s given us the platform for a streamlined and elegant disaster recovery solution, and the way it’s been delivered ensures business continuity also,” Keath said.