Scribd
Upload
467 views22 pages

Introduction to Hadoop and Big Data

This document provides an introduction to Apache Hadoop and big data. It discusses how Hadoop can help solve problems related to large-scale data storage, scalability, and analytics that have emerged since 2000. It provides examples of companies using Hadoop for applications like search, log processing, and recommendations. It also summarizes key aspects of Hadoop, including its use of HDFS for fault-tolerant storage, MapReduce for distributed processing, and an ecosystem of related projects. The document concludes with suggestions for how to get involved with Apache Hadoop and big data.

Uploaded by

indoos2000
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
467 views22 pages

Introduction to Hadoop and Big Data

This document provides an introduction to Apache Hadoop and big data. It discusses how Hadoop can help solve problems related to large-scale data storage, scalability, and analytics that have emerged since 2000. It provides examples of companies using Hadoop for applications like search, log processing, and recommendations. It also summarizes key aspects of Hadoop, including its use of HDFS for fault-tolerant storage, MapReduce for distributed processing, and an ecosystem of related projects. The document concludes with suggestions for how to get involved with Apache Hadoop and big data.

Uploaded by

indoos2000
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd

Big Data Camp, Delhi, Sep 10, 2011

Introduction to Hadoop / Big Data

Good Times < Year 2000


Online Applications- OLTP

Web Users Web Servers RDBMS

Analytics and Reporting- OLAP

Report Users Reporting Servers RDBMS DW

Year 2000 +
Online Applications- OLTP

Web Users

Web Servers

RDBMS

Analytics and Reporting- OLAP

Report Users Reporting Servers

RDBMS DW

Big Data- Problems to Solve

Storage

Fail
Scalability

The Knight in Shining Armor

Engine + Logic

File system

Video: What can Apache Hadoop Do for You?

Who Uses Hadoop?


Search Yahoo, Amazon, Zvents,

Log processing
Facebook, Yahoo Recommendation Systems

Facebook
Data Warehouse Facebook, AOL Video and Image Analysis New York Times, Eyealike INDIAN GOVERNMENT- UUID project

HDFS: Design Principles

Hardware will Fail!

Petabyte Scale Store!

HDFS: Design Principles

Map Reduce

Origin in Lisp!

Google- GFS paper!

Divide and Rule!

Map Reduce Programming Model

Borrows from functional programming

Users implement interlace of two functions :

map (in_key, in_value) ->

(out_key, intermediate value) list

reduce (out_key, intermediate value list) -> out_value list

Hadoop Map Reduce

Hadoop Map Reduce

Hadoop Example
Weather sensors collecting data every hour at many locations cross the globe gather a large volume of log data, which is a good candidate for analysis with MapReduce, since it is semistructured and recordoriented.
Data Format: The data is stored using a line-oriented ASCII format, in which each line is a record. The format supports a rich set of meteorological elements, many of which are optional or with variable data lengths. For simplicity, we shall focus on the basic elements, such as temperature, which are always present and are of fixed width.

Hadoop Example

Hadoop Example

Hadoop Ecosystem Map


1 Workflow 2 10
Cascading Cascading

12

Support More High Level Interfaces

Unstructured Data 6
5

High Level Interfaces 8 4


JAQL

13

Engine + Logic

File system 9

7 RDBMS

Structured Data
hiho

Monitor/manage 11 Hadoop ecosystem 14 OLTP

Java Applications

Sqoop

How can You Contribute?

Apache Hadoop Projects


Learn more about Hadoop Contribute to source code Participate in Mailing Lists/Forums Share blogs etc.

Impetus Open Source Projects


Github/Google code hosted projects Contribute to source code

Thank you
Visit [Link]

Big Data in EDW

20

Building Big Data Analytics Platform

Commercial

Open source

Hybrid

Teradata/ Netezza

CloverETL/ Kettle/ Talend

ETL - Open Source and Commercial

Greenplum/ Vertica/ Aster

Jaspersoft/ Pentaho Reporting Hadoop Apache Cassandra

Analytics - Open Source or Commercial

Informatica

Commercial Hadoop Versions

SAS/
Microstrategy/

Business Objects

Pentaho/ Jasper

Web Analytics

22

You might also like