Answer the big questions before you begin Big Data

August 29, 2016

I recently attended the first Big Data Wisconsin conference, which was part of Madison’s larger Forward Festival

With all kinds of arts, business, design and science activities going on all week, many of them free, Forward Festival really brings a lot of great energy to the city.

As a marketing professional and a consumer, I find “Big Data” fascinating. There are so many ways in which the knowledge we can gain from gathering data can improve lives from designing more effective chronic disease management programs to improving classroom performance. On the dark side, there are too many unresolved issues around privacy, who owns your data, and what companies can do with your data that doesn’t necessarily benefit you. 

I will confess that once terms like “Hadoop” and “Apache Kafka” started flying around, some of the talks were the equivalent of Gary Larson’s famous cartoon of what we say and what dogs hear. There was plenty of “Blah, blah, blah, Ginger” where, much like the cartoon dog, I just nodded my head without having a clue of what was being said.

However, what I heard over and over again (besides Hadoop), was a very fundamental issue with many development projects, whether the focus is designing a medical device, creating a service model or starting a Big Data initiative. It's that big, thorny sticking point: Why are you even doing this in the first place?

Dr. Jignesh Patel, a computer sciences professor at the University of Wisconsin-Madison and entrepreneur, noted during his presentation that this lack of fundamental prep work is common and leads to many Big Data initiatives that are a waste of time and money. A recent study by Forrester showed that 51 percent of companies with connected products are collecting Internet of Things (IOT) data, but most of them aren’t using it to provide any insights relating to customers

It's that big, thorny sticking point: Why are you even doing this in the first place?

Big Data is only as good as the value you extract from it for your business. What’s the driving business problem you’re trying to solve? As Patel said, the starting place for a Big Data initiative needs to start with the questions what, why and how. What, exactly, are you trying to accomplish by mining data? Why will it matter to your business — can you improve the user experience and/or monetize the results? How will you support this effort long-term? Can you train your internal staff or will you need to hire additional staff or outside consultants?

Knowing your what, why and how will save headaches and wasted effort down the road. To get real benefit from a Big Data program, Patel said it should be supported by a team using an Agile process that enables them to act upon incoming data every day in a Sprint-like approach.

That sounds a lot like iteration, which is key to how we design products at Design Concepts. Whether it’s data or a desk, designing meaningful solutions means asking those tough questions up front, listening to users and having a process in place to refine as you learn.

After you answer those fundamental questions, there are three components to any Big Data project:

  1. Data: Do you have the information you need? What’s missing? Where can you find it?
  2. Tools: What techniques, platforms and algorithms are necessary to extract the information you need?     
  3. Testing: What processes and methods will you use to analyze and act upon data?

Many companies decide that they need to jump on the Big Data bandwagon without thinking through the driving business problem and what types of data will actually be needed to answer the questions. Instead, they skip to step two—often leading to a suite of expensive technology with no crisply defined purpose. If you’ve seen the vast Kickstarter wasteland of abandoned connected product concepts, you know this isn’t a problem that’s confined to the land of IT. Sticking on a sensor or adding an app to a product doesn’t necessarily make it valuable to the user.

The third component, testing, should be carefully thought through before launching a Big Data initiative. Patel advised that it’s critical to have an A/B testing approach baked into the plan from the start. Testing needs to be iterative/agile/fast to get leanings quickly. If you're getting data every two weeks, "you're doing it wrong," Patel said. 

Finally, he ended his presentation with a plea for employers to invest in their current employees, who are more likely to have knowledge of the business but may lack technical skills. He pointed out that a great deal of technical training is available online at low to no cost. Fully utilizing the knowledgeable people you already have and providing them with new skills can not only improve the insights gleaned from Big Data – it can be a satisfier and retention tool for your best employees.