Identify a Big Data Problem
Do you really have a problem Big Data can solve?
The term Big Data is thrown around pretty loosely. Most people are aware of the term but it gets defined differently depending on who you are talking to. The truth is that most companies don’t have big data problems… just yet.
We feel there are actually fewer big data solutions right now than there could be. The main reason is that the business community hasn’t seen (or doesn’t know) how they can leverage big data for a true competitive advantage. When they can articulate their big data problems they want to solve, the tech community will surely rise to the occasion and solve them.
Another reason is that big data initiatives are inherently difficult to understand. Sometimes a lot of data is just that – a lot of data or bytes taking up space on servers. Putting it all together, analyzing the information, visualizing the data sets, and most importantly – getting useful information you can act on from all these efforts is no easy task. But that’s what Big Data truly is.
How big is a Big Data project?
Most companies have data sets that easily eclipse 100 GB, 1,000 GB, maybe even 100,000 GB? That’s a lot of information but rarely does it meet the litmus test for a true big data problem. Big Data information projects have these characteristics; large, diverse, distributed, and mixed data sets.
- Do you have petabyte of data? (That would be 1,048,576 GB’s) If no = then you don’t have a big data problem.
- Do you really need answers in real time? If no = then you probably don’t have a big data problem.
- Does it take a really long time (such as 8 hours or days) to calculate the results? If no, then you probably don’t have a big data problem.
- Are the answers that big data can provide really worth an investment of hundreds of thousands to millions of dollars to your business? If yes, then your business is a candidate for utilizing big data technologies.
Characteristics of Organizations that could utilize Big Data technologies;
- The organizations producing the data are big. Very big. Think Fortune 1000 or Government agencies. Most small and medium size organizations aren’t ready for big data projects.
- Producing data is an integral part of their daily activities.
- They have a massive amount of web users that produce information (such as tracking events).
- They track or produce enormous volumes of information like scientific, biomedical, and engineering research.
- They use equipment, sensors, and other automated transactions that create a new stream of data continuously.
- They really do need answers in real time for mission critical tasks that can only be determined via Big Data technologies.
- The large data sets are very diverse and/or very complex.
- They actually have a big enough footprint to get insight from social media, i.e. they are tweeted / discussed in social media circles.
- Their core business is to produce data and then sell the data sets and analytic tools around the data they created.
- They have the capital or sponsors willing to invest and support the big data initiative over time.
What do big data problems look like?
Much of the big data buzz is centered on mining social data like Twitter feeds, Facebook info, etc… It becomes a market researcher’s gold-mine after some really smart people come up with the right models to make sense of all the non-categorical information. So, now you can see how a certain topic or brand is trending. That’s nice. But what will you do with the answers big data provides???
Let’s say you know your market demographic cold. You would know what to offer an individual customer, on the fly, as they browse your online storefront. That could translate to real dollars. So how do I know all this information about my customers? Answer – Log file events. This is where you track each step of a user’s experience on a web site for example.
What does all that customer log file information allow you to do? Having hordes of log files on their own doesn’t really help you make decisions. Really smart people still need to take action on that information and do something with it. The next step is creating algorithms that allow you to more accurately ‘predict’ at what a customer would want to buy based on their current visit to your online storefront. Amazon put a lot of companies out of business by doing this better than anyone.
Why is there so much hype around Big Data?
Because if you are solving a big data problem that consumers think is valuable; then you have a fantastic competitive advantage over the competition.
Because there is more data being produced from machines, web sites, individuals, and other sources at the fastest pace in the history of mankind. We might as well put that data to use right?
Because scientists, engineers, and professors can use big data tools to solve increasingly complex problems and ever-evolving theories.
Big data problems are fantastic, challenging, and always changing.
Big Data Trends – our Magic 8 Ball says…
Big data technologies continue to advance, potential data sources grow at an exponential pace, and the cost of infrastructure has drastically been reduced to allow more organizations participate in solving big data problems.
We are in the infancy of solving big data problems, providing meaningful analytics, and empowering ordinary people so they can take advantage of the information.
People will continue to ask increasingly complex questions. Big data solutions will allow those questions to be answered. The organizations that move on these advances quickly will have competitive insights and built-in advantages over the competition.