Big Data is a "sexy" name, designed to impress, for what I like to call a data phenomenon. Or perhaps I should say phenomena because a single data point simply doesn't qualify. Big data is comprised of information with three primary, defining characteristics: first is the sheer volume, in other words having a lot of it; the second is it's variety, coming from many different sources and having many different types; third is the speed at which it is interpreted and analyzed. Having the first two components is utterly pointless without the third.
When the concept of big data first appeared, most businesses were operating under the misapprehension that it was simply a matter of collecting a lot of information and that by "somehow" analyzing it they would gain a competitive advantage. Unfortunately that simply complicated their lives and added the burden of keeping all of this information away from prying eyes. Owning the data confers no advantage; you have to be able to use it.
(But it should not describe your database) Having a big pile of unsorted, unstructured data stored away is basically useless. You will never be able to find anything except by the most fortuitous bit of luck. If you’re going to store it, you need a way to analyze it. Software known as ETL (Extract, Transform, and Load) is specifically designed to comb through all your collected data, discover and assign relationships between the disparate parts, and then inject it into a data warehouse in a manner that makes it easy for analysis and reporting. Now a particularly powerful Database Management System (irrespective of whether you’re using SQL on premise, or a cloud system like Azure, or you need the power of Hadoop), is using distributed storage and in-memory analysis to generate lists, reports, and finally, actionable insights.
Now you are using Big Data. Ultimately, the more data you have the more specific and accurate your results will be. That means any models you create or simulations that you derive from the data are more reliable. The immense amounts of money and time spent to collect all of your business data, to measure metrics, and to record all of it, can finally become useful. Looked at properly, you can literally predict the future.
Practical insights from your data can emerge in many ways. For example, it may take the form of a sudden realization that your aging workforce is preparing to retire faster than expected, and that you haven't hired enough people to replace them given the growth trend of your business. If business stayed exactly the way it is now, your current hiring practices might be perfectly adequate. Until you match up your revenue models with your staffing data, you wouldn’t see this issue.
Let’s look a little deeper at this example. Without the insights of Big Data, you might never spot that one innocuous department, responsible for a mere 4% of your total sales, which was at the beginning of experiencing a doubling-trend every two years. In five years, it could be 20-24% or more of your business. You could hire now, and be ready for that. That would provide time to get the new people up-to-speed, and garner the benefits of working with your top staff to learn their secrets before they retire. Unrecognized, you would find that five years down the road you are in crisis. That field has grown, and all the top candidates have already been hired by people with more foresight than you…
Perhaps you can't use it—the culture of your organization may not let you utilize that information in a profitable way. Experience tells us that companies with a strong evidence-based decision-making culture do very well with Big Data. The best results are derived when:
Coaching is available to decision makers at all levels.
There is one definitive source from which they agree to obtain performance statistics.
They have clear business rules, but they're not afraid to update them as circumstances warrant and information becomes available.
They quickly provide feedback to decision makers.
It is a significantly difficult culture-change to make for a company where high-level executives make plans based on their "feelings" and "experience". How many high-power C_Os do you know that actually possess the humility to let statistics speak, and facts overrule intuition? They're hired for their expertise and unless it is their idea to benefit from Big Data, then it is a threat to their power. The benefits can be amazing, but it takes a pretty amazing executive to actually make it work.
If you have a culture ruled by evidence-based decision-making, you're going to do very well with Big Data. In a top-down command structure, you're going to have to rely on the skill and expertise of your decision makers. If their names happen to be Elon and Richard, you will probably be okay.
For the rest, competing against Big Data, you're going to have to be at the very top of your form.
Check out our post about things to consider if you are considering implementing Hadoop.