This post is brought to you by Tim Spooner. Managing Director at Consumer Cloud
The amount of data in our world has been exploding. Companies capture trillions of bytes of information about their customers, suppliers, and operations, and millions of networked sensors are being embedded in the physical world in devices such as mobile phones and automobiles, sensing, creating, and communicating data. Multimedia and individuals with smartphones and on social network sites will continue to fuel exponential growth.
However, in line with this explosion of data, the ability to store, aggregate, and combine data and then use the results to perform deep analyses has become ever more accessible. General advancements in computing processing speed, memory capacity, its equivalent in digital storage developments, and cloud computing continue to lower costs and other technology barriers. For less than $600, an individual can purchase a disk drive with the capacity to store all of the world’s music. Last, the means to extract insight from data are also markedly improving as software available to apply increasingly sophisticated techniques combines with growing computing horsepower.
The interest in “Big Data”is given importance because it potentially provides a new, more cost effective and efficient way of identifying insight that enables improved business decision-making and performance.But many of the considerations and learning’s from the past decades are as applicable, if not more so, to Big Data. Therefore the brands that use these well-established principles to decide whether to use Big Data systems andtools or not, on what to use them and how best to leverage the learning from them will benefit the most.
So what are the key considerations and challenges that organisations face when considering whether to invest in Big Data projects?
- Advanced Analysis Techniques Don’t Mean Advanced Analysis; Analysis for analysis sake is clearly pointless. Companies need to think carefully about whether analysis is required and what the benefits are likely to be before undertaking any analysis. Benefits accrue from analysis that delivers insight that can deliver performance improvements and/or competitive advantage. Whilst data is a critical ingredient, the most effective approach to deliver meaningful insight is to identify the business opportunity and then determining how analysis and modelling can help improve performance.
- Garbage In Garbage Out; There is an unfortunate tendency for Big Data fans and technology providers to almost position the approach as a panacea and imply that it somehow removes the need to think about the data being loaded, it’s accuracy and format. The term “unstructured” is imprecise and doesn’t account for the many varying and subtle structures typically associated with Big Data types.
Big Data may well have different data types within the same set that do not contain the same structure and therefore, Big Data is probably better termed “multi-structured” as it could include text strings, documents of all types, audio and video files, metadata, web pages, email messages, social media feeds, form data, and so on. The consistent trait of these varied data types is that the data schema isn’t known or defined when the data is captured and stored. But don’t assume that it does away with the need to do additional work to prepare the data first – it is well accepted for traditional analytics and data mining projects that 80% of the analysis time is spent not analysing but preparing the data. Big Data practitioners already consistently report that 80% of the effort involved in dealing with data is cleaning it up in the first place, as Pete Warden observes in his Big Data Glossary: “I probably spend more time turning messy source data into something usable than I do on the rest of the data analysis process combined.”
Moreover, when analysing data it is important to understand how the data was generated in order to understand whether it is actually useful. For example, one might expect whether or not a consumer has subscribed for an e-newsletter to help explain their relative level of engagement with a brand – but if on checking how the data was collected we establish that certain consumers were automatically registered then this data is unlikely to be useful in analysis. Information in its raw form is too diffuse to be usable. It must be refined and concentrated – much like crude oil.
- Does your Company naturally collect all customer purchase moments?; Many companies cannot realistically take advantage of sophisticated analysis, Big Data or not, because they simply don’t have access to information on the majority of customers and their purchasing. Amazon knows exactly who its customers are, what they buy, when and how much they pay for it as a natural bi-product of the way they do business. But, it is a fact that many companies simply don’t have this information and therefore much of their potential uses for Big Data will be limited to either performance improvements or non-purchase related interactions with the brand. The Database Marketing, Relationship Marketing and CRM pioneers either missed or chose to ignore this key fact and to date much of the Big Data movement seems to be falling into the same trap.
- Consolidation of Multiple Data Sets; For many years now those companies which have understood the need to think creatively and cast the net as wide as possible to source data to support analysis and insight projects have generally benefited more from information. This requires not just looking at and merging multiple data sets from within an organisations existing information systems but at external data sources. One of the potentially most significant changes that have facilitated the need for and potential interest in Big Data is the plethora of new data sources such as social media, machine sensors and publically available data collected and made available on the internet. When planning Big Data analysis projects companies need to think carefully about what data is likely to be useful and then identify what they have and where (and if) they can source any missing data from external sources. Clearly if key data is missing and either cannot be sourced or it would cost too much to do so then this should impact the decision as to whether or not to proceed with any analysis in the first place.
- Ensure Insight Can be Leveraged; Almost as much thought and effort needs to go into planning how to implement the insights as in setting up the analysis project in the first place. This requires a realistic assessment of delivery capabilities and most notably ensuring that whilst the models themselves may be quite sophisticated their outputs need to be relatively simple to understand and use for the staff that will be expected to work with them. It is therefore generally a good idea to separate the full day-to-day responsibility for implementation from the analysts/data scientists that built the models. Visual displays and dashboards are generally a good way to deliver the information to front line managers and staff.
Of equal importance is the need to align people’s job roles and performance appraisal with the required objectives. Just as siloed data hampers the ability to derive benefit from data insight projects (including Big Data) so can traditional functional based organisational structures if key objectives aren’t aligned and managed across functional areas. Again, this challenge is not new. Consider for example, a Sales Call Centre where one key metric for personnel is almost always how quickly they can finish the call and how many sales per hour/day/week they can make. But from an insight point of view it may be critical to understand what led to the sale. However, asking Sales focused Call Centre staff to spend an extra few minutes per call to collect this information accurately invariably fails. No matter how much training and education is undertaken this isn’t likely to change. What is required is a change to the measurement process linked to their job description and performance evaluation – in many cases this even needs to be integrated into their remuneration package.
- Big Data analysis will more often than not tell you what is happening but not necessarily why; companies should be prepared to spend a potentially significant amount of effort in understanding the output of certain analysis projects. In some cases it may be necessary to conduct specific research to understand why a sequence of events is happening.
- Are You Really Ready for Big Data; Key factors influencing the likely success of any Big Data initiatives are corporate culture and existing expertise.
Benefiting from Big Data means investing in teams with the relevant skillset – Data Scientists. Good Data Scientists need to combine mathematical strengths, programming and scientific instinct as well as, increasingly, a strong commercial awareness. These experts need to be given sufficient freedom to explore a number of different hypotheses but be integrated into the business as a whole. Companies will critically need to surround them with an organisational willingness to understand and use data for advantage.
As mentioned already, careful consideration needs to be given to aligning individuals and departments roles & responsibilities and remuneration with companywide objectives.
There is also the consideration of IT and changing traditional views and values regarding where a company’s data sits. It is almost guaranteed that in order to take advantage of Big Data solutions and manage requirements and delivery over time in a cost-effective and efficient manner that organisations will need to leverage cloud based services in some way shape or form. This means a company being prepared and able to have their key customer and transactional information residing in physical storage outside of their organisation – yes it can be secured/protected, as indeed it must be, but many companies (and particularly their IT departments) have often had a far more myopic view to where their key company data resides and why.
And, perhaps most importantly, the push needs to come from senior executives who need to be prepared to switch from decisions made based purely on past experience to one which is led by the key insights delivered from the data.
In all analysis, whether Big Data or not, it is important to make sure you are realistic in your assessment of where you are as an organisation in relation to delivering meaningful insight from data. Ensure you have buy-in and real support at a senior level and invest in the right types of people. But most importantly, don’t assume every analysis project is necessarily a Big Data one and don’t assume that you have the right data to deliver meaningful insight.
Views on Customer Insight 