Three Big Data Trends To Watch in 2015

All new technologies go through an adoption cycle that waxes and wanes along with the attendant hype. Some technologies never make it through the so called “trough of disillusionment,” the phase in the cycle of technology adoption (as defined by analyst firm Gartner) when interest wanes.

With hundreds of technology firms pursuing the Big Data opportunity, there is certainly a lot of hype. But in my opinion, 2015 will be the year when Big Data makes the transition from hyperbole to productivity in the form of Applied Big Data solutions, particularly within the enterprise market.

Applied Big Data solutions—applications that are pre-built with with Big Data approaches that solve specific sets of business challenges—focus on helping non-technical people gain actionable insights so they can make evidence-based decisions.

Here are the top three trends to watch in the year ahead.

1. Applications Will Help Users Delve into the “Why” of Results.

When working with data, people can usually anticipate their initial questions. The key to a successful analytic solution is to design the application so that the users can go anywhere they need from the initial insight, guided by their sense of discovery. It is also essential that the application allows for an easy method of sharing the discovery with others that need to know, so that they also can satisfy their curiosity.

Here’s an example that illustrates the need to think about data causally:

A company is located in a community with a highly diverse minority population. Trying to reflect the community in which they operate, they are focused on hiring minorities. But while they constantly hire minorities, their overall minority representation does not improve.

In reality, the right course of action was to refine their employee retention efforts, and not focus entirely on hiring. The majority of this organization’s departments have been very successful in retaining and promoting their minority members. But this fact was hidden in the average calculations that were skewed by a small number of departments which were dramatically failing in their retention objectives. A well thought-out analytic application would have pointed out the failing outliers and identified the root causes of the failure.

The challenge that prevents today’s Big Data solutions to answer the second order questions is that the solution designers are technologists who usually do not understand the business user’s needs or thought processes.

It is this challenge that must be addressed, and we expect to see in 2015 increasing focus on Applied Big Data applications that are designed to answer the business user domain questions at a level of insight that is built on the understanding of the business content, as well as the relationship between data sources and data elements.

2. Innovation Will Be Driven By Smaller Screen Real Estate.

The technology that has made the biggest difference in the last five years is the smart mobile platform. Think tablets, smartphones and, potentially, wearable devices such as iWatch. IDC predicts that 8.5 million wearable devices will ship into Canada through 2018, with 111 million expected to ship globally.

Data visualization will need to focus on delivering precise, intelligent alerts for the smaller screen real estate. A great example for illustration is the car dashboard. While we drive, we get constant data flow from the gauges on the dashboard about speed or remaining fuel. But the real critical information—such as low fuel condition, road icing, engine overheating, low tire pressure, or brake line failure—is delivered through warning lights.

Alerts are particularly useful outcomes from Big Data analytics and well suited for the mobile platform. In the future, wearable devices will provide a useful means for the display of contextual alerts while the user is engaged in everyday tasks.

Clearly, wearable devices are still in their very early stages and mass appeal applications are yet to be invented. However, smartphones and tablets are everywhere, and increasingly used as the preferred or even the only data access point.

In 2015, we expect the mobile device to drive an acceleration of the wholesale re-engineering of most human-computer interfaces from the traditional page-format driven by form design to a dashboard with gauges, graphics, video and warning lights, and from asynchronous (eg. documents, email) to nearly real-time (eg, SMS, tweets) interactions. The context in which users access the data will increase in importance.

3. User Expectations Will Continue to Rise.

In the business realm, for users to get value out of their data, they typically must take a series of long, complex, technically challenging, risk prone and expensive steps using Business Intelligence and Big Data tools. To date, the Big Data vendors have focused on solving massive data storage and fast retrieval challenges. They have yet to create useful tools that allow business people to ask questions and receive answers without the employment of extremely skilled “data scientists.” A lot of nuance is lost in translation and lead-time between the business user’s questions and the data scientists’ output.

Clearly, there are cases where the deployment of a data scientist will result in useful insight from Big Data. However, these cases are generally not scalable due to cost and leadtime to delivery. Yet, virtually every business user has a need to deeply understand and continually improve the business process for which they are responsible.

Increasingly in 2015 and beyond, we expect that business people will demand answers to questions derived from a deep understanding of historical and projected data without the need to depend on IT or data scientists to interpret their data. The market is eager for such solutions.

Business users need to move beyond the hype of Big Data to getting help with answering their business questions. With cloud based Applied Big Data solutions, the customer may not care whether these applications have a Hadoop file system under the hood, or use approaches like in-memory data and MapReduce. But for technology providers, Applied Big Data represents a tremendous opportunity to build solutions that marry the power, speed and scalability of Big Data with domain expertise and industry best-practices (domain questions, metrics, navigation, visualization)—and as a result fundamentally change the way organizations use data to help them optimize their business.