Spark算子系列第1篇： reduceByKey 和 groupByKey

先通过示例代码来理解reduceByKey和groupByKey：

scala> val wordsRDD = sc.parallelize(Array("one", "two", "two", "three", "three", "three"),2).map(word => (word, 1))
wordsRDD: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[33] at map at <console>:24

scala> wordsRDD.collect
res37: Array[(String, Int)] = Array((one,1), (two,1), (two,1), (three,1), (three,1), (three,1))

scala> wordsRDD.foreach(x=>println(x._1+"|"+x._2))
one|1
three|1
three|1
two|1
three|1
two|1

//reduceByKey算子：
scala> wordsRDD.reduceByKey(_+_).foreach(x=>println(x._1+"|"+x._2))
two|2
one|1
three|3

scala> wordsRDD.reduceByKey(_+_)
res40: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[34] at reduceByKey at <console>:27

//groupByKey算子：
scala> wordsRDD.groupByKey().foreach(x=>println(x._1+"|"+x._2.sum))
two|2
one|1
three|3

scala> wordsRDD.groupByKey()
res41: org.apache.spark.rdd.RDD[(String, Iterable[Int])] = ShuffledRDD[35] at groupByKey at <console>:27

scala> wordsRDD.groupByKey().collect
res43: Array[(String, Iterable[Int])] = Array((two,CompactBuffer(1, 1)), (one,CompactBuffer(1)), (three,CompactBuffer(1, 1, 1)))

scala> wordsRDD.groupByKey().foreach(println)
(two,CompactBuffer(1, 1))
(one,CompactBuffer(1))
(three,CompactBuffer(1, 1, 1))

分析如下： 

1、从上面的运行结果可以看到，虽然两个函数都能得出正确的结果， 但reduceByKey函数更适合使用在大数据集上。

这是因为reduceByKey可以在每个分区移动数据之前将输出数据与一个共用的key结合，数据示例如下图所示：

2、当调用 groupByKey时，所有的键值对(key-value pair) 都会被移动,在网络上传输这些数据非常没必要，数据示例如下图所示：

3、通过上面两个图，可以想象一个非常大的数据集，在使用 reduceByKey 和 groupByKey 时他们的差别会被放大更多倍。另外，当移动的数据量大于单台执行机器内存总量时，Spark还需要把数据保存到磁盘上，这会更加影响性能。因此避免使用 GroupByKey。

算子源代码解析：

1、reduceByKey源码

 /**
   * Merge the values for each key using an associative and commutative reduce function. This will
   * also perform the merging locally on each mapper before sending results to a reducer, similarly
   * to a "combiner" in MapReduce.
   */
  def reduceByKey(partitioner: Partitioner, func: (V, V) => V): RDD[(K, V)] = self.withScope {
    combineByKeyWithClassTag[V]((v: V) => v, func, func, partitioner)
  }

  /**
   * Merge the values for each key using an associative and commutative reduce function. This will
   * also perform the merging locally on each mapper before sendi