Five Things That Happened To Data In 2015

2015 Was yet another fascinating and challenging year for all those involved with data. In this summary David Reed reveals his top 5 significant data happenings.

1 - Charities exposed

Good deeds can't hide bad practices. That was the hard lesson learnt this year by charities when a tsunami of media coverage led to a government-backed enquiry and fresh regulation. What started with the tragic suicide of a pensioner spiraled into the Etherington report and its chief fix for charity data problems, the Fundraising Preference Service.

Few should regret the impact on the "dark DM" data brokers and call centres that for years have traded consumer names with permissions long past their sell-by date. All should wonder at the scale of the practice - the ICO has written to over 1,000 such companies it has identified as operating at the margins of compliance (or beyond).

It remains to be seen how the mechanics of the newly regulated charity data eco-system will function. Fundraisers will have to work hard to rebuild trust at a time when data controllers in every sector will be trying to do the same thing. British consumer-like to give - but they don't like being taken for granted.

2 - Election results called wrong
Not even David Cameron expected the outcome of May's general election - the Prime Minister had reportedly not written an acceptance speech, let alone a note of thanks for winning the first workable Conservative majority since 1992.

For market researchers it was a humiliating fail - but big data analysts, including the much-heralded Nate Silver, former social media guru to President Obama, were caught out just as badly. Qualitative and quantitative measures of voting intention all called the wrong result, placing a big question mark over the value of predictive analytics of all types.

The lesson to be learned is that human behaviour is still not capable of being accurately modeled by even the smartest machines. People lie (including to themselves), they change their minds and, crucially, they act on impulse. Apart from voters still committed to tribal party allegiances, the "don't knows" acted as an unknown variable in the model. Think about that the next time you present an insight with a high confidence rating and consider whether you have excluded or accurately weighted the consumer base about whose intentions you know the least.

The only group that got a lot of these predictions correct was CASM part of DEMOS who are connected to Kings College London and Sussex University in the UK who have a software analysis programme called Method 52 which accurately analyses Social Media.

3 - Safe Harbour unsafe
US-based companies have taken it for granted that they will be able to ship data on their EU-based customers back to the homeland. The Schrems court case put a sizeable blockade in their way, at least in principle for now. Safe Harbour still applies to retrospectively-transferred data, but a new fix is needed going forward.

Three strategies are being adopted in the wake of the judgement: carry on as before and see what negotiations on Safe Harbour 2.0 end up demanding; build data centres inside the EU so data does not need to be shipped; or ignore the whole problem and carry on doing what you wanted to in the first place.

The first of these is unlikely to result in any regulatory penalties for now as it is recognised that the lawmakers had not kept up with other demands, (principally the requirement of US security services to access all the data they want without putting in place the few safeguards which European citizens enjoy).

The second is more sustainable and is helping to fuel both capacity and capability around data management across the EU. As for the third, it is not only out of step with the law, it is also not in line with the mood of consumers who have been spooked by the way their data can be examined by the spooks. For now, consumers are still willing to accept the US data homing model, but perhaps only until real alternatives emerge.

4 - Tesco data sale fail
Tesco started 2015 looking around for some quick cash. It's attention turned to dunnhumby, the data asset, which it had inexplicably been ignoring for some time (how else to explain the retailer's failure to spot shifts in customers' buying behaviours?) For a while it looked as if it might have a £2 billion windfall on its books.

Then prospective buyers started their due diligence. What would they get for their money? Rights to Clubcard data? No, they would stay firmly under the grocery chain's control. Cutting-edge big data algorithms and IP? No, there was little sign of that inside the family-sized analytics business. Instead, new owners would gain some reasonably smart people and a few years contracting to Tesco before facing a competitive pitch.

 One insider's best guess at the true value was of revenues of around £10 million a year, putting any valuation a long way off what was being asked.

So what had gone wrong with former innovator and once triumphant data operation? A combination of incumbent apathy and constrained specifications from its major client appear to have stopped the company from leaping into the new world of big data analytics, or even securing core assets in the same way as its deadly rival, Nectar. For now, Tesco and dunnhumby are stuck with each other in a marriage that doesn't appear to be making either of them happy.

5. - Big data booms
Predictions of the growth of big data as a genuine focus for investment by companies have not been hard to come by in 2015. More difficult have been real-world examples of projects emerging from proof of concept stage and delivering actual business value.

But make no mistake Hadoop clusters are being spun up all around UK plc. and ever more companies are circling their data wagons around Apache Spark. Big data solutions are becoming the new normal, especially as firms discover just how much cost they can save by migrating out of conventional data warehousing environments. It may not be possible to name practitioners right now, but the days of the big stack technology solution are starting to look numbered.

DataIQ: http://bit.ly/1PRJGdQ

David Reed is the editor of DataIQ