The Problem With Big Data Is That Nobody Understands It

Illustration for article titled The Problem With Big Data Is That Nobody Understands It

The concept of Big Data—the practice of acquiring, analyzing and interpreting ridiculously huge data sets—is something much of the technology and business world is extremely excited about. But excited is about as far as it goes because, currently, there just aren't enough practitioners to make it work.

While there's much fuss made about the endless possibilities of what can be done with the streams of data garnered from the web, the hard truth is that few people know what to do with it. The Wall Street Journal points to a report by McKinsey, which explains:

"A significant constraint on realizing value from Big Data will be a shortage of talent, particularly of people with deep expertise in statistics and machine learning, and the managers and analysts who know how to operate companies by using insights from Big Data. We project a need for 1.5 million additional managers and analysts in the United States who can ask the right questions and consume the results of the analysis of Big Data effectively."


But finding those people might not be very easy. Hilary Mason—inspirational chief scientists for, one of Fortune's 40 Under 40 to Watch and one of the leading proponents of Big Data—explains to the Wall Street Journal that people who work successfully in the sector have a rare combination of skills:

"They can take a data set and model it mathematically and understand the math required to build those models; they can actually do that, which means they have the engineering skills…and finally they are someone who can find insights and tell stories from their data. That means asking the right questions, and that is usually the hardest piece."

The way to fix that? As with any emerging field, there needs to be a change in the way we think about the sector, and the way we train people to enter it. Pat Gelsinger, president and chief operating officer of EMC Corphe, a giant US data company, explained to the Journal:

"IBM started a generation of Cobol programmers. Thirty years ago we didn't have computer-science departments; now every quality school on the planet has a CS department. Now nobody has a data-science department; in 30 years every school on the planet will have one."


That may not be all that's required, of course, but with the concept of Big Data refusing to go away, we'll have to work the rest out as we go. [Wall Street Journal]

Image by ARTSILENSEcom/Shutterstock


Share This Story

Get our newsletter


I find this report surprising. There is a degree known as Integrated Systems Engineering (formerly Industrial and Systems Engineering or just Industrial Engineering) that does this. You're taught enough about statistics, mechanical engineering, computer science, economics, psychology, ergonomics, and business that you could be a good fit for this position. Namely, taking data sets, creating a mathematical formulas to model the data, finding correlations, making hypothesis, and testing those hypotheses.

People with these degrees have a wide range of jobs. I know some that analyze large amounts of data to find patterns in people applying for loans at banks; some analyze production line data to determine where waste, errors, and cost can be reduced (traditional industrial engineering work); some work for theme parks to determine how to make ride lines move more quickly and how to occupy people in line so that they're more willing to wait; some work for fast food companies to determine the best way to bring in customers and get them through the line at the lowest cost with the least amount of waste and error.

I could go on and on about what they do but the underlying theme is that they work with large amounts of data to pull out a cause and effect that might not be easily visible.