A day in the life of a virtual data recovery expert

A day in the life of a virtual data recovery expert:

6:37 pm - An emergency call is received by the company’s external data recovery company team. Their client is a large financial services organisation that was in the middle of upgrading storage space and something went terribly wrong. The virtual servers contained on the original storage volume were deleted. Three days of important client data was lost in SQL databases contained within the virtual servers.

6.40 pm - A data recovery engineer is quickly added to the call and I discuss the options available to recover the data. As time is critical for the client (the data needs to be available the next morning), we suggest using a Remote Data Recovery (RDR) service. This service allows the engineer to connect remotely via an encrypted peer- to-peer connection - very secure and no client data is ever transmitted across the internet. After a lengthy consultation with the engineer, a plan of attack is agreed on and the client signs off on the evaluation stage of the recovery.

7.05 pm - The IT staff work with the data recovery company’s staff to establish a remote connection. A proprietary client is downloaded from the data recovery company’s website that connects to an RDR server in Australia. Once the connection is established, a RAM drive is created and all necessary data recovery tools are uploaded for the engineer to work with. The engineer needs to see each hard drive on an individual basis and scans are conducted to test the data integrity. If there are any signs of hardware failure, the drives need to be shipped to the data recovery organisation’s cleanroom for mechanical repair.

7.15 pm - The engineer starts the scanning process going for data structures. All work is being processed at the client premises and only screen shots and key strokes are transmitted over the internet. No client data leaves the data centre.

9.37 pm - The Australian-based engineers hand off the recovery to the London-located engineers who will work with it until the US team comes online later in their day.

Midnight - The UK team provides an update report to me that I then relay to the client. He is happy to know that the file system can be recovered along with the virtual machines. The recovery will be complex as there are layers to complete before the SQL databases can be extracted and the client’s records recovered. Reports should be completed by 1 am and the client can then make an informed decision about whether to proceed with the recovery or not, depending on the files able to be recovered, the quality of the data and the cost involved. I know the client is not going to purchase corrupted data and by presenting reports that show the files that are able to be recovered, we can prove that it is faster, cost effective and more efficient than re-keying the last three days of entries into the database.

1.00 am - The client is happy to hear the reports are completed and looks through to make sure that all the relevant files have been found. After further discussion with the engineers and myself he is keen to go ahead with the recovery. The engineers request that his IT team connects another storage volume to his server and the process of copying out the data commences. The client signs off on the data recovery quotation.

3:37 am - The copy out of the virtual machines (VMs) is completed. A total of 500 Gb of data has been recovered from the deleted logical unit number (LUN) on the SAN.

3.45 am - London has passed this information on to the US engineers. They start working with the VMs that have been copied out. Their aim is to copy out the SQL databases from the VMs so they can be mounted on other servers before the 7 am deadline. Reports are run to make sure that the SQL databases are not damaged. If areas are found, the engineers can pattern fill with characters to identify problem reports. At the moment it looks like no damage has been done.

5 am - The US engineers hand back to the Australian engineers and another conference call is held to discuss the outcome of the work completed by the virtual engineers overnight. The client is happy with the status report and the decision is taken to drop the connection so the client’s IT team can start the work of configuring access to the databases before staff need access at 7 am. Staff will, of course, be oblivious that any of the drama of the last 18 hours has occurred and that they could have been spending their day rekeying data.

*Adrian Briscoe, General Manager - Asia Pacific, Kroll Ontrack