Big Data in Practice

What do you think of when you think of "big data"?

For many, it's a nebulous term that invokes images of huge server farms humming
away. Or perhaps you think of receiving some kind of personalized advertisement
from a retailer.
But big data is so much deeper and broader than that. I believe there are 10 major
areas in which big data is currently being used to excellent advantage in practice -
but within those arenas, data can be put to almost any purpose.

1. Understanding and Targeting Customers

This is one of the biggest and most publicized areas of big data use today. Here, big
data is used to better understand customers and their behaviors and preferences.
Companies are keen to expand their traditional data sets with social media data,
browser logs as well as text analytics and sensor data to get a more complete picture
of their customers. The big objective, in many cases, is to create predictive models.
You might remember the example of U.S. retailer Target, who is now able to very
accurately predict when one of their customers will expect a baby. Using big data,
Telecom companies can now better predict customer churn; Wal-Mart can predict
what products will sell; and car insurance companies understand how well their
customers actually drive.
Ski resorts are even using data to understand and target their patrons. RFID tags
inserted into lift tickets can cut back on fraud and wait times at the lifts, as well as
help ski resorts understand traffic patterns, which lifts and runs are most popular at
which times of day, and even help track the movements of an individual skier if he
were to become lost.
Imagine being an avid skier and receiving customized invitations from your favorite
resort when there's fresh powder on your favorite run, or text alerts letting you know
when the lift lines are shortest. They've also taken the data to the people, providing
websites and apps that will display your day's stats, from how many runs you
slalomed to how many vertical feet you traversed, which you can then share on social
media or use to compete with family and friends.
Even government election campaigns can be optimised using big data analytics.
Some believe Obama's win after the 2012 presidential election campaign was due to
his team's superior ability to use big data analytics.
2. Understanding and Optimising Business Processes
Big data is also increasingly used to optimise business processes. Retailers are able
to optimise their stock based on predictions generated from social media data, web
search trends and weather forecasts.
One particular business process that is seeing a lot of big data analytics is supply
chain or delivery route optimisation. Here, geographic positioning and radio
frequency identification sensors are used to track goods or delivery vehicles and
optimise routes by integrating live traffic data, etc. HR business processes are also
being improved using big data analytics.
This includes the optimisation of talent acquisition - Moneyball style - as well as the
measurement of company culture and staff engagement using big data tools. For
example, one company, Sociometric Solutions, puts sensors into employee name
badges that can detect social dynamics in the workplace. The sensors report on how
employees move around the workplace, with whom they speak, and even the tone of
voice they use when communicating.
One of the company's clients, Bank of America, noticed that its top performing
employees at call centers were those who took breaks together. They instituted
group break policies and performance improved 23 percent.
You may have seen the RFID tags you can attach to things like your phone, your keys,
or your glasses, which can then help you locate those things when they inevitably get
lost. But suppose you could take that technology to the next level and create smart
labels that could stick on practically anything. Plus, they can tell you a lot more than
just where a thing is; they can tell you its temperature, the moisture level, whether or
not it's moving, and more.
Suddenly, this unlocks a whole new realm of "small data;" if big data is looking at
vast quantities of information and analysing it for patterns, then small data is about
looking at the data for an individual product - say, a container of yogurt in a shipment
- and being able to know if it's likely to go off before it reaches the store.
This part of the Internet of Things holds incredible promise for improving everything
from logistics to health care, and I believe we're still just on the cusp of
understanding what this incredible technology can do - as when electricity was only
used to power light bulbs.
3. Personal Quantification and Performance Optimisation
Big data is not just for companies and governments but also for all of us individually.
We can now benefit from the data generated from wearable devices such as smart
watches or smart bracelets. Take the Up band from Jawbone as an example: the
armband collects data on our calorie consumption, activity levels, and our sleep
patterns. While it gives individuals rich insights, the real value is in analysing the
collective data.
In Jawbone's case, the company now collects 60 years worth of sleep data every
night. Analysing such volumes of data will bring entirely new insights that it can feed
back to individual users.
The other area where we benefit from big data analytics is finding love - online this
is. Most online dating sites apply big data tools and algorithms to find us the most
appropriate matches.
4. Improving Healthcare and Public Health
The computing power of big data analytics enables us to decode entire DNA strings
in minutes and will allow us to find new cures and better understand and predict
disease patterns. Just think of what happens when all the individual data from smart
watches and wearable devices can be used to apply it to millions of people and their
various diseases. The clinical trials of the future won't be limited by small sample
sizes but could potentially include everyone!
Apple's new health app, called ResearchKit, has effectively just turned your phone
into a biomedical research device. Researchers can now create studies through
which they collect data and input from users phones to compile data for health
studies. Your phone might track how many steps you take in a day, or prompt you to
answer questions about how you feel after your chemo, or how your Parkinson's
disease is progressing. It's hoped that making the process easier and more automatic
will dramatically increase the number of participants a study can attract as well as
the fidelity of the data.
Big data techniques are already being used to monitor babies in a specialist
premature and sick baby unit. By recording and analysing every heartbeat and
breathing pattern of every baby, the unit was able to develop algorithms that can
now predict infections 24 hours before any physical symptoms appear. That way, the
team can intervene early and save fragile babies in an environment where every hour
counts.
What's more, big data analytics allow us to monitor and predict the developments of
epidemics and disease outbreaks. Integrating data from medical records with social
media analytics enables us to monitor flu outbreaks in real-time, simply by listening
to what people are saying, i.e. "Feeling rubbish today - in bed with a cold".
Of course, while much has been made in the past of Google's ability to predict flu
outbreaks based on search traffic, their model didn't work in 2014. Google itself
admits that just because you search for "flu symptoms," it doesn't mean you're sick.
5. Improving Sports Performance
Most elite sports have now embraced big data analytics. We have the IBM
SlamTracker tool for tennis tournaments; we use video analytics that track the
performance of every player in a football or baseball game, and sensor technology in
sports equipment such as basket balls or golf clubs allows us to get feedback (via
smart phones and cloud servers) on our game and how to improve it. Many elite
sports teams also track athletes outside of the sporting environment - using smart
technology to track nutrition and sleep, as well as social media conversations to
monitor emotional wellbeing.
The NFL has developed its own platform of applications to assist all 32 teams in
making the best decisions based on everything from the condition of the grass on the
field, to the weather, to statistics about an individual player's performance while in
university. It is all in the name of strategy as well as reducing player injuries.
One of the really cool new things I have come across is a smart yoga mat: sensors
embedded in the mat will be able to provide feedback on your postures, score your
practice, and even guide you through an at-home practice.
6. Improving Science and Research
Science and research is currently being transformed by the new possibilities big data
brings. Take, for example, CERN, the nuclear physics lab with its Large Hadron
Collider, the world's largest and most powerful particle accelerator. Experiments to
unlock the secrets of our universe - how it started and works - generate huge
amounts of data.
The CERN data center has 65,000 processors to analyse its 30 petabytes of data.
However, it uses the computing powers of thousands of computers distributed
across 150 data centers worldwide to analyse the data. Such computing powers can
be leveraged to transform so many other areas of science and research.
The computing power of big data could also be applied to any set of data, opening
up new sources to scientists. Census data and other government collected data can
more easily be accessed and analysed by researchers to create bigger and better
pictures of our health and social sciences.
7. Optimising Machine and Device Performance
Big data analytics help machines and devices become smarter and more
autonomous. For example, big data tools are used to operate Google's self-driving
car. The Toyota Prius is fitted with cameras, GPS as well as powerful computers and
sensors to safely drive on the road without the intervention of human beings. We can
even use big data tools to optimise the performance of computers and data
warehouses.
Xcel Energy initiated one of the first ever tests of a " smart grid" in Boulder, Colorado,
installing smart meters on customers' homes that would allow them to log into a
website and see their energy usage in real time. The smart grid would also
theoretically allow power companies to predict usage in order to plan for future
infrastructure needs and prevent brown out scenarios. In Ireland, grocery chain
Tescos has its warehouse employees wear armbands that track the goods they take
from the shelves, distributes tasks, and even forecasts completion time for a job.
8. Improving Security and Law Enforcement.
Big data is applied heavily in improving security and enabling law enforcement. I am
sure you are aware of the revelations that the National Security Agency (NSA) in the
U.S. uses big data analytics to foil terrorist plots (and maybe spy on us). Others use
big data techniques to detect and prevent cyber attacks. Police forces use big data
tools to catch criminals and even predict criminal activity and credit card companies
use big data use it to detect fraudulent transactions.
In February 2014, the Chicago Police Department sent uniformed officers to make
"custom notification" visits to individuals they had identified as likely to commit a
crime through a computer generated list. The idea was to prevent crime by providing
certain individuals with information about job training programs, or let them know
about increased penalties for people with certain backgrounds. But many community
groups cried foul and called the practice profiling.
9. Improving and Optimising Cities and Countries
Big data is used to improve many aspects of our cities and countries. For example, it
allows cities to optimise traffic flows based on real time traffic information as well as
social media and weather data. A number of cities are currently piloting big data
analytics with the aim of turning themselves into Smart Cities, where the transport
infrastructure and utility processes are all joined up. Where a bus would wait for a
delayed train and where traffic signals predict traffic volumes and operate to
minimise jams.
The city of Long Beach, California is using smart water meters to detect illegal
watering in real time and have been used to help some homeowners cut their water
usage by as much as 80 percent. That's vital when the state is going through its
worst drought in recorded history and the governor has enacted the first-ever state-
wide water restrictions.
Los Angeles uses data from magnetic road sensors and traffic cameras to control
traffic lights and thus the flow (or congestion) of traffic around the city. The
computerised system controls 4,500 traffic signals around the city and has reduced
traffic congestion by an estimated 16 percent.
A tech startup called Veniam is testing a new way to create mobile wi-fi hotspots all
over the city in Porto, Portugal. More than 600 city buses and taxis have been
equipped with wifi transmitters, creating the largest free wi-fi hotspot in the world.
Veniam sells the routers and service to the city, which in turn provides the wi-fi free
to citizens, like a public utility. In exchange, the city gets an enormous amount of data
- with the idea being that the data can be used to offset the cost of the wi-fi in other
areas. For example, in Porto, sensors tell the city's waste management department
when dumpsters are full, so they don't waste time, man hours, or fuel emptying
containers that are only partly full.
10. Financial Trading
My final category of big data application comes from financial trading. High-
Frequency Trading (HFT) is an area where big data finds a lot of use today. Here, big
data algorithms are used to make trading decisions. Today, the majority of equity
trading now takes place via data algorithms that increasingly take into account
signals from social media networks and news websites to make, buy and sell
decisions in split seconds.
Computers are programmed with complex algorithms that scan markets for a set of
customisable conditions and search for trading opportunities. The programs can be
designed to work with no human interaction or with human interaction, depending on
the needs and desires of the client.
The most sophisticated of these programs are now also designed to change as
markets change, rather than being hardcoded.
For me, the 10 categories I have outlined here represent the areas in which big data
is applied the most. Of course there are so many other applications of big data and
there will be many new categories as the tools become more widespread.
You might also be interested in reading my latest case study collection: Big Data
Case Study Collection: 7 Amazing Companies That Really Get Big Data as well as my
book Big Data: Using SMART Big Data, Analytics and Metrics To Make Better
Decisions and Improve Performance
Where to go from here
If you would like to know more about Hadoop or other big data tools, check out my
articles on:
 How Big Data Is Transforming Every Business, In Every Industry
Whats Really Behind The Big Data Hype?
And free eBooks:
Beyond The Big Data Buzz: How Data Is Disrupting Every Business In Every
Industry In The World
Big Data Case Study Collection: 7 Amazing Companies That Really Get Big Data
Or browse the Big Data section of this site to find more articles and many practical
examples.

Written by

Bernard Marr
Bernard Marr is a bestselling author, keynote speaker, and advisor to companies and governments. He has worked with and advised many of the world's best-known
organisations. LinkedIn has recently ranked Bernard as one of the top 10 Business Influencers in the world (in fact, No 5 - just behind Bill Gates and Richard Branson). He
writes on the topics of intelligent business performance for various publications including Forbes, HuffPost, and LinkedIn Pulse. His blogs and SlideShare presentation
have millions of readers.