You may have come across the term Big Data, Big Data means data sets so large and/or complex that our normal data processing applications can’t handle it.

To analyse, search, share, store or transfer, even to capture, curate or visualize such masses of information requires enormous processing power if it is to be of value to the gatherer.

So who are the the gatherers, well they are first and foremost Governments, large Corporations, Scientists for research, Businesses and finance, Media and Advertising, and Meteorology, complex environmental research for example.

What is more,

it has become relatively easy and cheap to collect from all sorts of devices, mobile, aerial, photographs, microphones, the radio frequency ID in your credit cards and wireless sensor networks, software logs.

Because the world’s technological per-capita capacity to store information has roughly doubled every 3 years since the 1980s the amount of storage requirements is truly mind blowing even for a smaller business, and the control and who should own big data initiatives across an organization can become more than a headache.

Desktop statistics and visualization packages for relational database management systems often have difficulty handling Big Data as it requires massively parallel software running on tens, hundreds or even thousands of servers, think Google or NSA.

We don’t have to worry about our PC size Data, our PCs can handle it – mostly – because we have had an equal enormous increase in the size of our PC’s memory, but still we buy external hard drives to cope with our images and spiralling emails. I started on my first computer with 48kb memory, no not Gigabytes, but kilobytes, now I have 4 external hard drives in addition to my computer’s terabyte and they are all filling fast, so then we use the cloud and that too is ever increasing.

Big Data is a big deal and it is growing and the truth is that whilst we understand the the value of data most organizations and Governments don’t know how to use that data to its full potential. Perhaps that is a good thing.

On the US Government’s data sharing website there is 200,000 datasets from 170 sources to help provide information to the public for scrutiny, but even then we are but at the edge of understanding what this data can do and more publicly available data may be needed to help it to truly reach its potential.

Think of the massive changes already made in the manufacturing industries where sensors are providing ever more important data to achieve productivity and quality enhancements and the data obtained proving more valuable than the machines themselves. Think ‘The Internet of Things’ where sensors provide data for all sort of information from shopping to security to your car’s performance.

We have faith in technology to solve our problems, to allow us to make smarter decisions, better and more informed ones. Our medical people can make better diagnoses, better appropriate treatments, our sports selectors can pick the best players, and ultimately transparency may root out corrupt officials and Politicians.

But, irrespective of how much information we collect, another horrific Paris event may occur because in the end it is not just about how much information we collect, maybe it is about the people who work with that data.

You may have read recently about another nearly 2000 people being recruited in the UK by the GCHQ, what we might term the Security setup, because without the people on the ground and in local communities the gathering of intelligence will not suffice from technology alone since the data is useless without  the human element to connect the dots. Data is neutral.

With other words, data cannot do something for you unless you know how to make the connections and there is still a fair way to go until the computers can extract meaning for us.

Technology is a tool, we can improve and enhance our own capabilities by making the best use of computers and the technology man has created, and even in the military it is acknowledged the the most effective strategy moving forward is to team the troops with machines.

There is still a danger that whilst the technological change is increasing more rapidly we still tend to overestimate our own idea of how advanced we are.

Let us make full use of technology we understand to help us to be smarter, healthier and safer, as long as we remember that it is a means to an end, not the end itself.

So with all the tech devices likely to be bought as Christmas present can we now look forward to a smarter, more knowledgeable, happier and healthier population for 2016, I do hope so.

Wishing you, dear Reader, a safe celebration and a happy holiday.