Don't let little data become a big hassle

Little data may become more valuable than big data. But how do you gather it efficiently and securely, and what should you do with it?

Has big data had its day? A growing number of businesses and pundits alike are promoting the value of ‘little data’ - sets of information that are limited in scope to a specific person, organisation or device.

Some see it as a means by which to rapidly boost personal productivity; others, as delivering the same insights as big data but at a lesser (or, if you’re an optimist, ‘more targeted’) scale. Both are right - to an extent.

What both viewpoints leave out is that little data is a subset of big data, rather than being a straight-up alternative. That means the same guiding principles for intelligent, efficient big data handling - automation, clear business goals and the twin gatekeepers of privacy and security - also apply to its smaller sibling.

In fact, many individuals and enterprises are already in a position to gather little data. The bigger question is whether and why they’d want to do so.

A trivial solution

The data that we need to monitor performance and productivity is, in most circumstances, already available to us. Let’s say, for example, that you’re in a sales role and you want to figure out the efficiency of your cold-calling routine. As with a big data strategy, you’d identify a hypothesis (perhaps that people stay on the line longer at certain times of the day) and figure out what data would prove or disprove it (the average length of your calls, segmented by hour).

But unlike with big data, this information is easily accessible: via, say, a VNQM (VoIP Network Quality Manager) tool to track packet traffic and performance.

These little data ‘platforms’ are embedded throughout the average enterprise IT set-up. If you’re looking to track your most procrastination-likely hours, a network traffic analyser can track which apps you’re using at different times of the day; a user device tracker can find out how often you check your phone during those riveting all-hands meetings. So the information is there - it’s just a question how you analyse it, and why you would want to.

Little Brother is watching

This is where the privacy/security question rears its binary head, even more so than when discussing big data’s applications in the enterprise sphere. Because little data by definition covers very discrete data sets, and because its objectives are mostly related to individual performance, the potential for very targeted and very damaging breaches of confidentiality is extremely high. And while the systems mentioned above don’t collect the details of this personal information - such as, for example, what you say on a VoIP call - they can still paint a potentially embarrassing or even libellous picture of individuals or organisations if given into the wrong hands.

There are numerous security and de-identification methods which can reduce this threat, including many with which system and network admins will already be familiar. De-identification is already a major part of businesses working in sensitive areas, like the legal and financial services industries. But enterprises should be asking themselves: does the risk need to be taken?

Little data can, of course, also assist in improving areas like CRM analysis or internal process monitoring - but by this time, you’re starting to look at problems which demand big data or machine data solutions, where limiting your scope of analysis is also going to limit your ability to come up with effective solutions.

Any enterprise considering the use of little data to improve employees’ performance should think about the privacy trade-off very carefully. They should also mull over what we like to think of as the Rule of Online Forms: if someone suspects their privacy might be at risk, they’ll just falsify their data. Add to that the likelihood of individuals or groups ‘gaming’ their metrics to look better at appraisal time and you have a data concoction which is both highly combustible and not particularly useful.

So is there a time when little data’s benefits might outweigh their costs?

Hacking your life

The answer is a definite maybe. For individuals looking to improve their own performance, analysing their own personal data stores can yield some extremely valuable insights. We collect more data about ourselves than ever before, and the rise of wearable devices like FitBit and Google Glass looks set to spur even more rapid growth in what we know about what we do.

Apps which can track and provide recommendations based on this data will, we think, grow in number and popularity as more people take up ‘life-hacking’ as a discipline. You don’t even have to look at biometric or behavioural data, either: an app which can analyse your calendar schedule could easily allot you different tasks to complete, based on the amount of free time you have to get them done.

Again, security and privacy will be concerns, but the big difference is that rather than a large organisation it’s the individual who has control - and who ultimately takes the blame for any breach. Little data is probably too big a risk for any organisation at this stage: better to stick with big data, particularly sets which are stripped of personal identifiers from the get-go (such as machine and sensor data), than risk the privacy quagmire of tracking employee ‘performance data’.

But as the amount of personal data continues to grow, it’s only a matter of time before people get wise to it - and hopefully turn little data into a force for self-betterment instead of a tool for more nefarious purposes. Calorie-shaming, anyone?

*Lawrence Garvin is a Head Geek and technical product marketing manager at SolarWinds. A highly experienced IT professional, Lawrence is a nine-time Microsoft MVP and a Microsoft Certified IT Professional.

^Thomas LaRock is a Head Geek at SolarWinds, bringing his expertise in database administration and SQL Server. He has over 15 years of IT industry experience as a programmer, developer, analyst and database administrator.

Image credit: ©iStockphoto.com/4X-image