1

MapReduce: Simpli_ed Data Processing on Large Clusters

The MapReduce programming model has been successfully used at Google for many different purposes. We attribute this success to several reasons.
First, the model is easy to use, even for programmers without experience with parallel and distributed systems, since it hides the details of parallelization, fault-tolerance, locality optimization, and load balancing. Second, a large variety of problems are easily expressible as MapReduce computations. For example, MapReduce is used for the generation of data for Google's production web search service, for sorting, for data mining, for machine learning, and many other systems. Third, we have developed an implementation of MapReduce that scales to large clusters of machines comprising thousands of machines. The implementation makes ef_cient use of these machine resources and therefore is suitable for use on many of the large computational problems encountered at Google.

We have learned several things from this work.

First, restricting the programming model makes it easy to parallelize and distribute computations and to make suchcomputations fault-tolerant. Second, network bandwidth is a scarce resource. A number of optimizations in our system are therefore targeted at reducing the amount of data sent across the network: the locality optimization allows us to read data from local disks, and writing a single copy of the intermediate data to local disk saves network bandwidth. Third, redundant execution can be used to reduce the impact of slow machines, and to handle machine failures and data loss. # MapReduce reads its input from and writes its output to the Google File System. # The users of MapReduce within Google's engineering organization for providing helpful feedback, suggestions, and bug reports. MapReduce is a programming model and an associated implementation for processing and generating large data set.

Examples
Here are a few simple example of interesting programs that can be easily expressed as MapReduce computations. Distributed Grep: The map function emits a line if it matches a supplied pattern. The reduce function is an identity function that just copies the supplied intermediate data to the output. Distributed Sort:The map function extracts the key from each record,and emits a pair. The reduce function emits all pairs unchanged.This computation depends on the partitioning facilities

Implementation
Many different implementations of the MapReduce interface are possible.The right choice depends on the environment. For example,one implementation may be suitable for a small shared-memory machine, another for a large NUMA multi-processor, and yet another for an even larger collection of networked machines.

Describes an implementation targeted to the computing environment in wide use at Google:

In our environment:
(1) Machines are typically dual-processor x86 processors running Linux, with 2-4 GB of memory per machine. (2) Commodity networking hardware is used . typically either 100 megabits/second or 1 gigabit/second at the machine level, but averaging considerably less in overall bisection bandwidth. (3) A cluster consists of hundreds or thousands of machines. (4) Storage is provided by inexpensive IDE disks attached directly to individual machines. (5) Users submit jobs to a scheduling system. Each job consists of a set of tasks, and is mapped by the scheduler to a set of available machines within a cluster.

Execution Overview
1- The Map invocations are distributed across multiple machines by automatically partitioning the input data into a set of M splits. 2- The input splits can be processed in parallel by different machines. Reduce invocations are distributed by partitioning the intermediate key space into R pieces using a partitioning function (e.g., hash (key) mod R). 3- The number of partitions (R) and the partitioning function are specified by the user. 3- Figure 1 shows the overall flow of a MapReduce operation in our implementation. When the user program calls the MapReduce function, the following sequence of actions occurs : 1. The MapReduce library in the user program first splits the input files into M pieces of 16 megabytes to 64 megabytes (MB) per piece. It then starts up many copies of the program on a cluster of machines. 2. One of the copies of the program is special - the master. The rest are workers that are assigned work by the master. There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task. 3. A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory. 4. Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function. The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers. 5. When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers. When a reduce worker has read all intermediate data, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task. If the amount of intermediate data is too large to fit in memory, an external sort is used. 6. The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user's Reduce function.The output of the Reduce function is appended to a final output file for this reduce partition. 7. When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point,the MapReduce call in the user program returns back to the user code.

After successful completion

1-The output of the mapreduce execution is available in the R output files (one per reduce task, with file names as specified by the user). 2- Users do not need to combine these R output files into one file . they often pass these files as input to another MapReduce call, or use them from another distributed application that is able to deal with input that is partitioned into multiple files.

Fault Tolerance
Since the MapReduce library is designed to help process very large amounts of data using hundreds or thousands of machines, the library must tolerate machine failures gracefully.

Combiner Function
The Combiner function is executed on each machine that performs a map task. The same code is used to implement both the combiner and the reduce functions.

The only difference between a reduce function and a combiner function is how the
MapReduce library handles the output of the function. The output of a reduce function is written to the final output file. The output of a combiner function is written to an intermediate file that will be sent to a reduce task. Partial combining significantly speeds up certain classes of MapReduce operations. Appendix A contains an example that uses a combiner.

Performance
we measure the performance of MapReduce on two computations running on a large cluster of machines.

1-Cluster Configuration
All of the programs were executed on a cluster that consisted of approximately 1800 machines.

2- Grep
The grep program scans through 1010 100-byte records, searching for a relatively rare threecharacter pattern (the pattern occurs in 92,337 records). The input is split into approximately 64MB pieces (M = 15000), and the entire output is placed in one file (R = 1).

3- Sort
The sort program sorts 1010 100-byte records (approximately 1 terabyte of data). This program is modeled after the TeraSort benchmark .The sorting program consists of less than 50 lines of user code. A three-line Map function extracts a 10-byte sorting key from a text line and emits the key and the original text line as the intermediate key/value pair.