We are now creating 2.5 quintillion bytes of data per day - so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few.
This data is big data.
To put this further into perspective; the US Library of Congress in April 2011 possessed a huge 235 terabytes of data in April 2011. But 15 out of 17 sectors in the United States have more data stored per company than the US Library of Congress.
In 2010, Gartner reported that enterprise data growth will be 650 percent over the next five years, and 80 percent of that will be unstructured.
So what's contributing to this explosion? The rising velocity, volume and variety of information captured by enterprises, the rise of multimedia and the digitisation of everything will continue to fuel exponential data growth for the foreseeable future.
Big Data presents Big Opportunities as well as Big Challenges.
According to a recent study, 58% of organisations surveyed say that big data is a challenge, 73% consider driving operational efficiencies to be the biggest benefit of a big data strategy and 50% say that big data helps in better meeting consumer demand and facilitating growth.
How can businesses leverage big data to unlock value? Big data can unlock significant value by making information transparent and usable at a much higher frequency as organizations create and store more transactional data in digital form, thereby allowing businesses to collect more accurate and detailed performance information, offer much more precisely tailored products or services through an ever-narrower segmentation of customers, substantially improve decision-making and improve the development of the next generation of products and services.
Instead of demographics, big data enables smart companies to future focus on psychographics - the values, beliefs and attitudes of its customers. For examples, MasterCard has formed a business intelligence unit that uses MasterCard's exhaust-data - its customer purchasing data - for research and business advice. Southwest Airlines uses real-time data and social analytics to boost customer service and give their customers tailored offers, and Williams-Sonoma uses psychometric segmentation based on big data to market in a tailored fashion to its database, boosting response rates by 10-18 times. Insurance companies are now using RFID devices fitted to cars, to provide real-time data feedback on driver behaviours and changing premiums as a result. London buses come fitted with smart LCD screens that change advertisements based on which psychographic neighbourhood they drive through.
For Netflix, it was so important to make good movie recommendations based on customer viewing data that the company sponsored a $1 million contest in 2009 for an algorithm that would improve recommendations by at least 10 percent.
For a hospital, it was so important to minimize unnecessary hospitalizations that it sponsored a $3 million contest for an algorithm that would analyze patient data to predict hospitalizations, so appropriate care plans could be implemented and hospitalizations averted.
For Amazon, it was so important to minimize costly returns that it created a set of algorithms, patented in December 2010, for letting recipients convert an undesired gift to a desired one before the gift is sent.
According to the McKinsey Global Institute, a retailer using big data to the full could increase its operating margin by more than 60 percent. Harnessing big data in the public sector has enormous potential, too. If US healthcare were to use big data creatively and effectively to drive efficiency and quality, the sector could create more than $300 billion in value every year. Two-thirds of that would be in the form of reducing US healthcare expenditure by about 8 percent. In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues.
This explosion in data is more than simply a matter of size; Data today is less about 1s and 0s and more about the story behind the numbers. Companies who use data cleverly and effectively, learn how to tell contextual and relevant stories, whether they use that customer data internally, or choose to on-sell it externally (staying privacy compliant of course) will be the ones to stand out as thought leaders over the coming years.
For innovators and thought leaders, big data presents a big opportunity to find insights in structured and unstructured types of data and content, to unlock value and answer questions that were previously considered beyond reach. Until now, there was no practical way to harvest this opportunity. Today, advanced analytics for big data uses state of the art technologies to open the door to a world of possibilities.
If you want to learn more about what opportunities and challenges big data presents for your industry, why not engage a futurist scenario planning exercise for your leaders?
*by Anders Sorman-Nilsson
research by Ware Kuo