“The modern age has a false sense of security because of the great mass of data at its disposal. But the valid issue is the extent to which man knows how to form and master the material at his command.”
You could be forgiven for thinking this quote was penned recently by one of any of the chief luminaries of our day when warning about the perils of assuming that just by having lots of data stored on a massively parallel software solution running on tens, hundreds, or even thousands of servers somewhere in the cloud means you’ll make smarter business decisions for the benefit of the bottom line. And indeed they would be right to be trying to get the message across that simply collecting, storing and analysing Big Data sets won’t necessarily lead to any insight let alone commercial advantage or betterment of the world in which we live. Worse still, without careful consideration and rigorous assessment to identify true insight it is all too easy to find misleading patterns in the data and mistake correlation with causation – the scope for finding misleading insights by definition increase given the sheer scale and coverage of the data being analysed.
But perhaps the most interesting thing to me about this quote is that it is attributed to none other than Johann Wolfgang Goethe and when……………………………………………….. in 1832! The problem has been the same for a good many years now. Whilst the scale may have changed the challenges remain the same – it’s how we master the opportunity and leverage the mass of data at our disposal that will determine whether we derive any real benefit.
Statisticians have spent centuries working tirelessly to understand what problems lie ahead and how these can be solved or at least minimised. With more data, faster processing and analysis tools all at a lower cost per x it would be a grave mistake to assume that this either solves the problems or renders the theory irrelevant. It has not and will not.
“There are a lot of small data problems that occur in big data,” says Spiegelhalter. “They don’t disappear because you’ve got lots of the stuff. They get worse.”
In business, as increasingly in our personal lives, we have become a quick fix society. We buy ready cooked meals rather than going to all the hassle of cooking – and if we can’t even be bothered to go to the supermarket to pick up a ready meal and then pop it into the oven then we get a takeaway or eat out at one of the ever increasing number of cafes and restaurant chains. In business we increasingly seem to look for the quick, hassle free ways to make our working days easier and seem to have confused the potential power of computers and the ever multiplying data being generated as being some sort of magic wand that will make instant, profitable and flawless business changing decisions. Whereas in fact without stopping to ask the right questions, collect the right data, shape it in the right way, analyse it in a robust and meaningful manner and then consider the why’s to the what’s all this does is increase the speed and sheer scale of the disasters we can unleash on the business. Remember, to err is divine but to really screw things up you need a computer!