Click Frenzy, the cloud and capacity planning

There are constant headlines about the dire straits Australian retail finds itself in. In some cases, it is competition from overseas online stores that is to blame, although the issues are far more complex and involve a clear shift in the way many people shop. Online stores aren’t just about prices - they are about variety and service as well.

In response to this, and mirroring the success of the USA’s ‘Black Friday’ sales, many Australian retailers banded together to create Click Frenzy - a one-day, online shopping ‘event’ that was meant to bring together a bunch of very special one-off deals. What we’re interested in is the technology behind this and what we can learn from the system’s failure.

Click Frenzy was scheduled to start at 7pm EDST. Within just a few minutes it was clear that the site was doomed. The site was flooded by shoppers, who created their own DDoS attack (distributed denial of shopping according to a humorous tweeter) and the infrastructure and systems ground to a halt. This is despite the helpful message posted on the Click Frenzy FAQ page:

Will your servers crash?

We sure hope not! We know there will be enormous volumes of visitors during Click Frenzy, particularly during the early part of the event, but we have taken every precaution to ensure our servers will not go down, and we have advised our retailers of the traffic volumes they should expect. We are so excited to be sharing in the first ever Frenzy with you! You won’t just be shopping online, you’ll be making history!!

Despite the hyperbole about “making history”, it’s clear that the operators of Click Frenzy were expecting heavy traffic.

Click Frenzy lists as it technology partners Magneto eCommerce and UltraServe. Clearly, they were using the cloud but weren’t able to ramp up services fast enough to match demand. The problems were exacerbated when many retail partner sites also capitulated to the traffic.

Cloud services, both infrastructure and software as a service (IaaS and SaaS respectively), can form a critical part of your infrastructure. Hybrid data centres, where your infrastructure is both inside and outside your wall, are clearly an important consideration for the future. The only way you can cost-effectively deliver an elastic infrastructure to move services that are subject to rapid changes is to use cloud services that you can escalate at times of high demand and reduce when things quieten down.

If the case of something like Click Frenzy, a one-day event, the required infrastructure for development can be kept lean for most of the year during development. In the lead-up to the event, when testing for both function and load management were in progress, the number of processors and memory, and the bandwidth could be increased in preparation for the event. Clearly, the operators of Click Frenzy either did not anticipate the traffic levels - hard to imagine given there was a registration process - or were unable to ramp services up fast enough.

Therein lies the second lesson. Do thorough analysis and ensure that your chosen service provider is able to keep up with the need for fast ramp-ups when you need them.

In the United Kingdom, daily newspaper The Guardian was able to do this successfully using Amazon’s cloud services. They ran a reader competition asking readers to review ministerial finance reports in order to assist with locating any alleged rorting. As reader interest peaked, they rapidly increased the infrastructure support. Once the initial interest fell back, they could pull their costs back. This is far more efficient and substantially more cost effective than deploying in-house systems.

If your business is reticent about using IaaS, one strategy can be to take a low-priority service, such as application development for a secondary system, and try it out. When it comes time for that system to go to production, use the cloud provider so that you can seriously road-test the service.

In the coming days, we will no doubt hear more about what went wrong with Click Frenzy. While it will be tempting to greet the analysis with a healthy dose of schadenfreude, pay close attention to the root causes. They may help you refine how you use your current cloud providers or assist with the selection process if you’re looking ahead.