The US consulate database crash - lessons to be learned

Most travellers who tried to get US travel papers processed last month know about the US consulate database crash that affected 200,000 or more visa-seekers worldwide. The culprit, a simple software patch, caused the US State Department’s central passport and visa system to crash on 19 July. While the data was backed up, the system was not, causing visa processing to grind to a halt during the slow recovery process.

There’s a data protection lesson from this situation that all companies can learn from: quick, reliable recovery depends on backup strategies that protect all company data, including configuration data and application code.

Expedited recovery

The Consular Consolidated Database, or CCD, is the central repository for all US visa applications and integrates with other federal intelligence databases. It’s one of the largest Oracle-based databases in the world, storing more than 75 million photos and 100 million visa files, according to The Register. During the outage, those files were inaccessible while IT restored the system. The downtime created a backlog of visa applications, causing headaches for those dependent on a visa for school, work or travel. The State Department's Consular Affairs office says it has made significant progress and that the majority of non-immigrant visas have been issued, yet applicants may still experience one-week delays on top of the normal processing time.

Even now, officials are still working through the application backlog. Some Chinese students studying abroad in the US likely won’t arrive in time for the start of their term. Irish citizens with flights booked for a vacation to the US may not receive their temporary visas in time. American parents who just adopted a baby are unable to get home with their new child. And the list goes on.

The good news is that the government didn’t lose the visa application data. The bad news is that recovery is slow because the system wasn’t backed up. The situation, and the lengthy recovery process, is a reminder that systems fail and accidents happen. Its job is to ensure the company has a data protection strategy in place to get back up and running quickly.

Downtime is expensive

New-generation data protection methods, such as image backup, speed up recovery times and give IT more control over company data. This is important given how expensive downtime can be. About 80% of IT professionals estimate that downtime costs their business at least $20,000 per hour and more than 20% estimate that it costs at least $100,000, according to IDC research.

In our data-driven world, companies are highly dependent on their data. To protect against data disasters or costly downtimes, companies need reliable protection technologies that cover all of the bases, including application, file and system configuration data. The State Department learned that lesson the hard way, but its experience reminds companies worldwide that recovery and backup must be a priority.

Image courtesy Angelo DeSantis under CC