分布式文件系统研究与应用
摘 要
分布式文件系统，在当今集群存储中起着重要的作用，其中 moosefs 更是当
中的佼佼者，他是一种分布式网络系统，它分布在多个计算机节点上面，每个节
点只会存储整个文件的一部分内容，它有多重备份，易于部署，方便扩容等诸多
的优点，而且它能够统一接口，使用它就像使用普通的文件系统一样，而且它对
小文件的处理，性能优秀。它采用当今主流的 Master /Chunk 设计，而且有强大
的日志功能，数据恢复更有保证。
本文所提及实现的是一个基于分布式文件系统的网盘，主要是通过大量的分
布式存储节点提供存储服务，再通过指定的服务器进行挂在到本地，对外提供存
储服务。提供统一的开放接口，提供给开发者使用。
此分布式网盘，可以提供数据自动备份，将数据备份为 3 份，备份到不同的
节点至上，从而使得数据的安全性大大的提高。
整个开放分布式网盘，主要由三个部分组成，Moosefs，分布式网盘，分布
式网盘开放接口。Moosefs 主要提供分布式存储，备份，冗余等功能，分布式网
盘主要提供逻辑上的操作，开放接口，是分布式网盘对外提供服务的桥梁，开发
者通过 API 来使用分布式网盘，从而提高分布式网盘的使用范围。
关键字: 分布式文件系统 Moosefs 数据恢复 开放式
The Design And Implememt of a Distribute Filesystem
Jia Xiaokang
(College of Informatics, South China Agricultural University, Guangzhou 510642, China)
Abstract: The distributed file system plays an important role in today’s clustered storage,
which moosefs is the leader among them, it is a distributed network system, it distribution in
multiple computer take nodes above, each node only will store the entirepart of the contents
of the file, it has multiple backup, easily deploy, convenient expansion, and it to unified
interface, use it just like ordinary file system,and its small file handing, performance is
excellent. It takes the Master/Chunk of today's mainstream design, and robust logging
functionality, data recovery is more assured.
This article is a reference implementation of a distributed is a network-base distributed
file system disk, primarily through a large number of distributed storage nodes providing
storage services, and then by the specified server hanging on to local , to provide storage
service.
This distributed network that provides automatic data backup, data backup 3, first
backup to a different node so that greatly improve data security.
Open distributed Web site as a whole, is composed of three key components, Moosefs,
distributed networks, distributed network of open interfaces. Moosefs primarily provide a
distributed storage, backup, redundant functions, operation on a distributed network disk
provides a logical, open interfaces, distributed Web site provides services that bridge
developer API to use a distributed network, so as to improve the use of distributed network
disk.
Key words: distributed file system; Moosefs; data recovery
目 录
1 引言..........................................................................................................................................6
1.1 课题背景与现状..................................................................................................................6
1.2 选题价值与意义..................................................................................................................6
1.3 研究内容..............................................................................................................................6
1.4 论文结构..............................................................................................................................6
2 分布式文件系统的介绍..........................................................................................................8
2.1 研究方法和技术路线..........................................................................................................8
2.2 Hadoop 中 HDFS 介绍........................................................................................................9
2.3 传统 NFS 文件系统介绍 ..................................................................................................12
2.4 闭源的 GFS 介绍 ..............................................................................................................12
2.5 开源的 TFS 介绍................................................................................................................13
2.6 较新 Moosefs 文件系统介绍............................................................................................14
2.7 各种分布式文件系统的概要分析....................................................................................17
3 文件系统的架构....................................................................................................................17
3.1 平台要求............................................................................................................................18
3.2 机器架构............................................................................................................................18
3.3 文件系统架构....................................................................................................................19
3.4 分布式文件系统的搭建....................................................................................................19
3.5 性能测试............................................................................................................................20
3.5.1 小文件性能测试..............................................................................................................21
4 开放网盘的实现方式............................................................................................................23
4.1 需求分析............................................................................................................................23
4.2 系统分析............................................................................................................................23
4.3 网盘的架构........................................................................................................................25
4.4 数据表结构........................................................................................................................27
4.5 核心代码实现....................................................................................................................28
5 测试用例................................................................................................................................33
5.1 用户认证............................................................................................................................34
5.2 上传下载............................................................................................................................36
5.3 获取目录信息....................................................................................................................38
5.4 数据恢复功能.....................................................................................................................39
6 总结与展望............................................................................................................................40
6.1 总结....................................................................................................................................40
6.2 展望....................................................................................................................................40
参 考 文 献..............................................................................................................................41
致 谢............................................................................................................................42
