big data challenges

1. Dealing with data growth

The most obvious challenge associated with big data is simply storing and
analyzing all that information. In its Digital Universe report, IDC estimates that the
amount of information stored in the world's IT systems is doubling about every two
years. By 2020, the total amount will be enough to fill a stack of tablets that
reaches from the earth to the moon 6.6 times. And enterprises have responsibility
or liability for about 85 percent of that information.

Much of that data is unstructured, meaning that it doesn't reside in a database.

Documents, photos, audio, videos and other unstructured data can be difficult to
search and analyze.

It's no surprise, then, that the IDG report found, "Managing unstructured data is
growing as a challenge – rising from 31 percent in 2015 to 45 percent in 2016."

In order to deal with data growth, organizations are turning to a number of

different technologies. When it comes to storage, converged and hyperconverged
infrastructure and software-defined storage can make it easier for companies to
scale their hardware. And technologies like compression, deduplication and tiering
can reduce the amount of space and the costs associated with big data storage.

On the management and analysis side, enterprises are using tools like NoSQL
databases, Hadoop, Spark, big data analytics software, business intelligence
applications, artificial intelligence and machine learning to help them comb
through their big data stores to find the insights their companies need.

2. Generating insights in a timely manner

Of course, organizations don't just want to store their big data — they want to use
that big data to achieve business goals. According to the NewVantage Partners
survey, the most common goals associated with big data projects included the
following:

1. Decreasing expenses through operational cost efficiencies

2. Establishing a data-driven culture
3. Creating new avenues for innovation and disruption
4. Accelerating the speed with which new capabilities and services are
deployed
 5. Launching new product and service offerings

All of those goals can help organizations become more competitive — but only if
they can extract insights from their big data and then act on those insights
quickly. PwC's Global Data and Analytics Survey 2016 found, "Everyone wants
decision-making to be faster, especially in banking, insurance, and healthcare."

To achieve that speed, some organizations are looking to a new generation of ETL
and analytics tools that dramatically reduce the time it takes to generate reports.
They are investing in software with real-time analytics capabilities that allows
them to respond to developments in the marketplace immediately.

3. Recruiting and retaining big data talent

But in order to develop, manage and run those applications that generate insights,
organizations need professionals with big data skills. That has driven up demand
for big data experts — and big data salaries have increased dramatically as a result.

The 2017 Robert Half Technology Salary Guide reported that big data engineers
were earning between $135,000 and $196,000 on average, while data scientist
salaries ranged from $116,000 to $163, 500. Even business intelligence analysts
were very well paid, making $118,000 to $138,750 per year.

In order to deal with talent shortages, organizations have a couple of options. First,
many are increasing their budgets and their recruitment and retention efforts.
Second, they are offering more training opportunities to their current staff
members in an attempt to develop the talent they need from within. Third, many
organizations are looking to technology. They are buying analytics solutions with
self-service and/or machine learning capabilities. Designed to be used by
professionals without a data science degree, these tools may help organizations
achieve their big data goals even if they do not have a lot of big data experts on
staff.

4. Integrating disparate data sources

The variety associated with big data leads to challenges in data integration. Big
data comes from a lot of different places — enterprise applications, social media
streams, email systems, employee-created documents, etc. Combining all that data
and reconciling it so that it can be used to create reports can be incredibly difficult.
Vendors offer a variety of ETL and data integration tools designed to make the
process easier, but many enterprises say that they have not solved the data
integration problem yet.

In response, many enterprises are turning to new technology solutions. In the IDG
report, 89 percent of those surveyed said that their companies planned to invest in
new big data tools in the next 12 to 18 months. When asked which kind of tools
they were planning to purchase, integration technology was second on the list,
behind data analytics software.

5. Validating data

Closely related to the idea of data integration is the idea of data validation. Often
organizations are getting similar pieces of data from different systems, and the data
in those different systems doesn't always agree. For example, the ecommerce
system may show daily sales at a certain level while the enterprise resource
planning (ERP) system has a slightly different number. Or a hospital's electronic
health record (EHR) system may have one address for a patient, while a partner
pharmacy has a different address on record.

The process of getting those records to agree, as well as making sure the records
are accurate, usable and secure, is called data governance. And in the AtScale 2016
Big Data Maturity Survey, the fastest-growing area of concern cited by
respondents was data governance.

Solving data governance challenges is very complex and is usually requires a

combination of policy changes and technology. Organizations often set up a group
of people to oversee data governance and write a set of policies and procedures.
They may also invest in data management solutions designed to simplify data
governance and help ensure the accuracy of big data stores — and the insights
derived from them.

6. Securing big data

Security is also a big concern for organizations with big data stores. After all, some
big data stores can be attractive targets for hackers or advanced persistent threats
(APTs).

However, most organizations seem to believe that their existing data security
methods are sufficient for their big data needs as well. In the IDG survey, less than
half of those surveyed (39 percent) said that they were using additional security
measure for their big data repositories or analyses. Among those who do use
additional measures, the most popular include identity and access control (59
percent), data encryption (52 percent) and data segregation (42 percent).

7. Organizational resistance

It is not only the technological aspects of big data that can be challenging —
people can be an issue too.

In the NewVantage Partners survey, 85.5 percent of those surveyed said that their
firms were committed to creating a data-driven culture, but only 37.1 percent said
they had been successful with those efforts. When asked about the impediments to
that culture shift, respondents pointed to three big obstacles within their
organizations:

 Insufficient organizational alignment (4.6 percent)

 Lack of middle management adoption and understanding (41.0 percent)
 Business resistance or lack of understanding (41.0 percent)

In order for organizations to capitalize on the opportunities offered by big data,

they are going to have to do some things differently. And that sort of change can be
tremendously difficult for large organizations.

The PwC report recommended, "To improve decision-making capabilities at your

company, you should continue to invest in strong leaders who understand data’s
possibilities and who will challenge the business."

One way to establish that sort of leadership is to appoint a chief data officer, a step
that NewVantage Partners said 55.9 percent of Fortune 1000 companies have
taken. But with or without a chief data officer, enterprises need executives,
directors and managers who are going to commit to overcoming their big data
challenges, if they want to remain competitive in the increasing data-driven
economy.