Commercial Analytics
of Clickstream Data using
Hadoop
June 2014
Submitted by:
Kartik Gupta
201100048
M.C.A
Thapar University
Submitted to:
School of Mathematics and
Computer Application
Department,
Thapar university,
Patiala.
Outline
Overview
Big Data
Hadoop
Major Steps
Results and Analysis
Conclusion and Future Scope
Overview
This Project gives an analytic report to find the
behavior and location of visitor using Hadoop.
Map Reduce is implemented to refine and sort
the raw data.
Searching is done based on the country, ip
addresses, Postal code, categories wise
Hadoop is a tool which converts the
unstructured, structured and semi-structured
data into pair into a single value which is
represented in binary format.
MapReduce framework is used for parallel
implementation.
Big Data
Big Data is a term used to describe large
collections of data that may be unstructured
grow so large and quickly that it is difficult to
manage with regular database or statistical
tools.
3 vs of Big data
Hadoop
Open source project started by Doug Cutting
A platform to manage Big Data
Helps in Distributed computing
Runs on Commodity Hardware
Data storage (HDFS)
Runs on commodity hardware (usually Linux)
Horizontally scalable
Processing (MapReduce)
Parallelized (scalable) processing
Fault Tolerant
CORE PARTS OF HADOOP
Hadoop Distributed File System(HDFS)
Hadoop Distributed File System (HDFS) is a
Java-based file system that provides scalable
and reliable data storage that is designed to
span large clusters of commodity servers.
Some specific features ensure that the
Hadoop clusters are highly functional
RackAwareness
Minimal Data Motion
Utilities
Rollback
Highly Operable
How HDFS works
MapReduce
MapReduce is a programming model and an
associated implementation for processing large
data sets.
MapReduce usually splits the input data-set into
independent chunks which are processed in a
completely parallel manner.
This allows programmers without any experience
with
parallel and distributed systems to easily
utilize the resources of a large distributed system.
The run-time system takes care of scheduling
tasks, monitoring them and re-executes the failed
tasks.
Execution flow in MapReduce
1. Mapreduce program that has been written tells
the job client to run a mapreduce job.
10
Execution flow in MapReduce
2.This sends a message to the Jobtracker which
produces a unique ID for the job.
11
Execution flow in MapReduce
3. JobClient copies job resources , such as jar file.
12
Execution flow in MapReduce
4. Once the resources are in Distributed
Filesystem, the JobClient can tell the JobTracker to
start the job.
13
Execution flow in MapReduce
5. The JobTracker does its own initialization for the
job.. It retrieves these input splits from the
distributed file system.
14
Execution flow in MapReduce
6. Now that the Jobtracker has work for
Tasktrackers, it will return the map task or reduce
task as response to the heart beat.
15
Execution flow in MapReduce
7. The TaskTracker need to obtain the code to
execute, so they get it from the shared file system.
16
Execution flow in MapReduce
8. The TaskTracker now will run the job.
17
OTHER TECHNOLOGICAL
TERMS
Clickstream Data
Clickstream data is an information trail a user leaves
behind while visiting a website. It is typically captured in
semi-structured website log files.
Potential Uses of Clickstream Data
What is the most efficient path for a site visitor to research
a product, and then buy it?
What products do visitors tend to buy together, and what
are they most likely to buy in the future?
Where should I spend resources on fixing or enhancing the
user experience on my website?
Basically we will focus on the path optimization use case.
Specifically: how can we improve our website to reduce
bounce rates and improve conversion?
STEP I
Upload Acme website log dataset contains about 4 million
rows of data, which represents five days of clickstream
data.
STEP II
Represent the dataset in unstructured format i.e
timestamp, registerd user swid, ip address,
geocoded ip address, url
STEP III
Represent the users data from the unstructured
loaddataset
STEP IV
Represent the products categories wise from
the dataset
STEP V
Shows the refine dataset of acme logfiles
STEP VI
Combine all the tables i.e acme log, products, users.
Results and Analysis
Configuration of Hadoop
Results and Analysis
Count the no of VISITORS from any country
Results and Analysis
Retrieving the ip address and displaying the state of visitors
Results and Analysis
Showing the no of ip access this category at a time
Results and Analysis
Initial stage of mapping and reduction
Results and Analysis
Category accessed by total no of ips
Results and Analysis
Showing shoes category acc to state access by total no of ip
Results and Analysis
showing details of ip accessed by visitors but gender wise
Result and Analysis
No of Females accessed this page
Result and Analysis
Total no of ip address accessed particular webpage
Result and Analysis
Calculate the sum of ages of all the visitors
Conclusion
The amount of clickstream data is rapidly
growing and with this demand for accessing
information
over
web
has
increased
significantly.
Therefore analyze the behavior and location
of the visitor.
It is inefficient to process large data using
traditional sequential method
Therefore MapReduce is used for processing
large datasets
Future Scope
Clickstream information play an important
role in a wide variety of applications such as
decision support systems, profile-based
marketing.
Location search is used by various industries
like telecom , e-commerce industry , in event
detection.
Nearest location method can be fused with
any other method to help in better way for
decision making.
Then the tradeoff would be done between
distance and other factor that would be fused
Thank you !!!
