Growing a distributed network with root and branch infrastructure management

A look at the evolution of out-of-band infrastructure management tools and technologies shows their growing impact beyond the data centre in managing today’s highly distributed networks.

Out-of-band management appliances are a staple of data centre infrastructure management.

They have played roles both as an access concentrator to simplify the management of high-density installs and as an essential remediation tool for troubleshooting and disaster recovery to minimise mean time to repair (MTTR) when systems lose in-band network connectivity.

Historically, the choice for out-of-band management was limited to either serial terminal server or KVM over IP (keyboard, video, mouse over internet protocol) switch.

At its most basic, the serial terminal server was a Cisco router repurposed as an access server, with additional RS232 async cards providing bullet-proof remote access to network router and switch management consoles, and the serial TTYs of UNIX and UNIX-like servers.

KVM over IP was reserved for servers running GUI-driven operating systems, chiefly Microsoft Windows, for remote point-and-click management when in-band management via Remote Desktop was not responding.

As a management technology, KVM always had its shortcomings. As an external bolt-on solution without direct access to the server’s system bus, even enterprise-grade KVM over IP switches suffered from poor video quality and mouse sync issues.

While modern servers can be specified with lights-out management cards that improve the user experience by serving virtual KVM directly via a built-in dedicated network port, the rapid growth of virtualisation has all but relegated KVM to the role of bit player in out-of-band server management.

In a virtualised environment, KVM over IP access to guest virtual machines is served by the hypervisor.

When there are server availability issues, out-of-band management is of the hypervisor itself - more often than not a UNIX-like system (such as Citrix XenServer as used by Amazon EC2 and Rackspace Cloud) and increasingly the Linux native Kernel-based Virtual Machine - and therefore better suited to command-line management via serial console.

In the meantime, the terminal servers of 10 years ago have evolved into today’s converged data centre infrastructure management appliances, providing out-of-band management of network, server and facilities infrastructure.

The first stage of this evolution was secure console servers, which bolstered terminal server remote access capabilities with the security features required by the modern data centre. 
This includes enterprise- and military-grade encryption of management traffic, granular user access control and integration with enterprise authentication systems like Active Directory and RADIUS two-factor.

Next was the integration of high-availability and resiliency features such as redundant power and ethernet and integrated PSTN or 3G cellular modem. This guarantees management availability even in the case of complete network meltdown.

Finally came the convergence of data centre facilities’ monitoring and management capabilities, such as power distribution and back-up power infrastructure and environmental monitoring.
Coupled with automatic notification and escalation of infrastructure fault conditions or environmental thresholds, this enables operators and administrators to respond to issues that may affect availability before they occur, proactively increasing mean time between failures (MTBF).

While the root of the network is currently well served by out-of-band management tools, it’s arguably in the branches of increasingly distributed installs where out-of-band infrastructure management is proving the most valuable.

There are three major trends driving this value proposition.

Firstly is the absolute reliance on always-up network connectivity. This is best illustrated by the distributed workforce, which now relies on the cloud for day-to-day systems like customer relationship management (CRM) and teleconferencing. For these sites, the cost of a network outage in lost productivity and opportunity has skyrocketed.

Secondly is the current economic climate, with tight budgets forcing businesses to do more with less. Cost-saving initiatives like cloud migration and staff reductions and centralisation have resulted in limited or no technical staff at remote sites. Once you add the costs of fuel, labour and downtime factored with the prolonged MTTR associated with a technician call-out to repair a remote network outage, these cutbacks can start to look like a false economy.

To operate effectively in this new environment, central technical staff needs new management tools.

This requirement has been met by the development of remote management appliances - essentially a converged infrastructure management appliance from the data centre in a scaled-down form factor.

The convergence of infrastructure management technologies means a single remote management appliance provides central staff with the ‘virtual remote hands’ to manage network, ICT and power - and even building management systems.

Environmental monitoring such as water leak and door open detection in distributed network cabinets act as ‘virtual eyes and ears’ for physical infrastructure security.

Finally, as copper PSTN lines become increasingly scarce, 3G cellular has proven a critical enabling technology for remote infrastructure management, serving secure remote access over an independent, high-speed network when the primary network link is down.