Spark2.2版本内核源码深度剖析（完整笔记）_hadoop伪分布式环境搭建资源-CSDN下载

共45个文件

html：27个

png：18个

5星 · 超过95%的资源需积分: 19 14 浏览量 2019-04-30 18:13:36 上传评论收藏 1.24MB ZIP 举报

资源推荐

资源详情

资源评论

收起资源包目录

Spark2.2版本内核源码深度剖析.zip （45个子文件）

3，基于Yarn的两种提交模式深度剖析.html 9KB

4，SparkContext原理剖析与源码分析.html 120KB

16-1，BlockManager原理剖析_files

BlockManager原理剖析.png 67KB

3，基于Yarn的两种提交模式深度剖析_files

基于YARN的两种提交模式深度剖析.png 55KB

Image.png 145KB

4，SparkContext原理剖析与源码分析_files

SparkContext原理剖析.png 44KB

18-1，Checkpoint原理剖析.html 68KB

0，目录入口：Spark内核源码深度剖析_index.html 3KB

6，Master注册机制原理剖析与源码分析.html 31KB

13，Executor原理剖析与源码分析.html 36KB

7，Master状态改变处理机制原理剖析与源码分析.html 59KB

14，Task原理剖析与源码分析_files

Task原理剖析.png 37KB

11-1，DAGScheduler原理剖析：stage划分算法原理剖析.html 8KB

11-2，DAGScheduler源码分析(stage划分算法、task最佳位置计算算法).html 155KB

15-2，优化后的Shuffle操作的原理剖析.html 3KB

5，Master主备切换机制原理剖析与源码分析_files

主备切换机制原理剖析.png 51KB

16-2，BlockManager源码剖析中.html 248KB

9，Worker原理剖析与源码分析.html 162KB

16-1，BlockManager原理剖析.html 4KB

18-2，Checkpoint源码剖析.html 35KB

5，Master主备切换机制原理剖析与源码分析.html 51KB

9，Worker原理剖析与源码分析_files

Worker原理剖析.png 31KB

1，Spark内核源码深度剖析.html 10KB

15-1，普通Shuffle操作的原理剖析.html 10KB

11-1，DAGScheduler原理剖析：stage划分算法原理剖析_files

Image.png 113KB

stage划分算法原理剖析.png 55KB

17-2，CacheManager源码剖析.html 198KB

17-1，CacheManager原理剖析_files

CacheManager原理剖析.png 43KB

2，宽依赖和窄依赖深度剖析.html 2KB

8，Master资源调度算法原理剖析与源码分析.html 52KB

15-3，Shuffle读写源码分析.html 168KB

17-1，CacheManager原理剖析.html 4KB

14，Task原理剖析与源码分析.html 110KB

15-1，普通Shuffle操作的原理剖析_files

普通的Shuffle操作原理剖析.png 62KB

1，Spark内核源码深度剖析_files

Spark内核架构深度剖析.png 78KB

2，宽依赖和窄依赖深度剖析_files

宽依赖和窄依赖深度剖析.png 56KB

10，job触发流程原理剖析与源码分析.html 57KB

15-2，优化后的Shuffle操作的原理剖析_files

优化后的Shuffle原理剖析.png 57KB

16-2，BlockManager源码剖析上.html 43KB

8，Master资源调度算法原理剖析与源码分析_files

Image.png 156KB

13，Executor原理剖析与源码分析_files

Executor原理剖析.png 23KB

18-1，Checkpoint原理剖析_files

Checkpoint原理剖析.png 69KB

12，TaskScheduler原理剖析与源码分析.html 161KB

6，Master注册机制原理剖析与源码分析_files

注册机制原理剖析.png 42KB

16-2，BlockManager源码剖析下.html 101KB

<html> <head> <title>16-2，BlockManager源码剖析中</title> <basefont face="微软雅黑" size="2" /> <meta http-equiv="Content-Type" content="text/html;charset=utf-8" /> <meta name="exporter-version" content="YXBJ Windows/600667 (zh-CN, DDL); Windows/10.0.0 (Win64);"/> <style> body, td { font-family: 微软雅黑; font-size: 10pt; } </style> </head> <body> <a name="3192"/> <h1>16-2，BlockManager源码剖析中</h1> <div> <span><div><div><div><div style="font-size: 10.8pt;"><span style="font-size: 11pt;">16-2，BlockManager源码剖析中</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; font-family: Consolas; font-weight: bold;">3，</span><span style="font-size: 11pt; font-family: Consolas; font-weight: bold;">BlockManager.scala</span></div><div><span style="font-size: 11pt; color: rgb(255, 0, 0);">/**</span></div><div><span style="font-size: 11pt; color: rgb(255, 0, 0);">* BlockManager运行在每个节点上，包括driver和executor，都会有一份，主要提供了在本地或者远程存取数据的功能</span></div><div><span style="font-size: 11pt; color: rgb(255, 0, 0);">* 支持内存、磁盘、堆外存储（Tychyon）</span></div><div><span style="font-size: 11pt; color: rgb(255, 0, 0); font-family: Consolas;">*/</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">private</span><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">[spark]</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">class</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">BlockManager(</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">executorId:</span> <span style="font-size: 10.8pt; color: rgb(32, 153, 157); font-family: Consolas;">String</span><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">rpcEnv: RpcEnv,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">master: BlockManagerMaster,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">serializerManager: SerializerManager,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">conf: SparkConf,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">memoryManager: MemoryManager,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">mapOutputTracker: MapOutputTracker,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">shuffleManager: ShuffleManager,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">blockTransferService: BlockTransferService,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">securityManager: SecurityManager,</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">numUsableCores: Int)</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">extends</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">BlockDataManager</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">with</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">BlockEvictionHandler</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">with</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">Logging {</span></div><div style="font-size: 10.8pt;"><br/></div><div style="background-color: rgb(255, 255, 255); font-size: 10.8pt;"><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">private</span><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">[spark]</span> <span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(102, 14, 122); font-family: Consolas; font-style: italic;">externalShuffleServiceEnabled</span> <span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">=</span></div><div style="background-color: rgb(255, 255, 255); font-size: 10.8pt;"><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">conf.getBoolean(</span><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 128, 0); font-family: Consolas; font-weight: bold;">"spark.shuffle.service.enabled"</span><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">,</span> <span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">false</span><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">)</span></div><div style="font-size: 10.8pt;"><br/></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="font-size: 10.8pt; color: rgb(102, 14, 122); font-family: Consolas; font-style: italic;">diskBlockManager</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">= {</span></div><div style="background-color: rgb(255, 255, 255); font-size: 10.8pt;"><span style="background-color: rgb(255, 255, 255); font-size: 10.8pt; color: rgb(128, 128, 128); font-family: Consolas; font-style: italic;">// Only perform cleanup if an external service is not serving our shuffle files.</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">val</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">deleteFilesOnStop =</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">!</span><span style="font-size: 10.8pt; color: rgb(102, 14, 122); font-family: Consolas; font-style: italic;">externalShuffleServiceEnabled</span> <span style="font-size: 10.8pt; color: rgb(0, 0, 0); font-family: Consolas;">|| executorId == SparkContext.</span><span style="font-size: 10.8pt; color: rgb(102, 14, 122); font-family: Consolas; font-style: italic;">DRIVER_IDENTIFIER</span></div><div style="font-size: 10.8pt;"><span style="font-size: 10.8pt; color: rgb(0, 0, 128); font-family: Consolas; font-weight: bold;">new</span> <span style="font-size: 10.8pt; color: rgb(

评论收藏

内容反馈