Distributed File System

1.
Lecture 2 –Distributed Filesystems 922EU3870 – Cloud Computing and Mobile Platforms, Autumn 2009 2009/9/21 Ping Yeh ( 葉平 ), Google, Inc.

2.
Outline Get toknow the numbers

3.
Filesystems overview

4.
Distributed file systemsBasic (example: NFS)

5.
Shared storage (example:Global FS)

6.
Wide-area (example: AFS)

7.
Fault-tolerant (example: Coda)

8.
Parallel (example: Lustre)

9.
Fault-tolerant and Parallel(example: dCache) The Google File System

10.
Homework

11.
Numbers real worldengineers should know L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 100 ns Main memory reference 100 ns Compress 1 KB with Zippy 10,000 ns Send 2 KB through 1 Gbps network 20,000 ns Read 1 MB sequentially from memory 250,000 ns Round trip within the same data center 500,000 ns Disk seek 10,000,000 ns Read 1 MB sequentially from network 10,000,000 ns Read 1 MB sequentially from disk 30,000,000 ns Round trip between California and Netherlands 150,000,000 ns

12.
The Joys ofReal Hardware Typical first year for a new cluster: ~0.5 overheating (power down most machines in <5 mins, ~1-2 days to recover) ~1 PDU failure (~500-1000 machines suddenly disappear, ~6 hours to come back) ~1 rack-move (plenty of warning, ~500-1000 machines powered down, ~6 hours) ~1 network rewiring (rolling ~5% of machines down over 2-day span) ~20 rack failures (40-80 machines instantly disappear, 1-6 hours to get back) ~5 racks go wonky (40-80 machines see 50% packetloss) ~8 network maintenances (4 might cause ~30-minute random connectivity losses) ~12 router reloads (takes out DNS and external vips for a couple minutes) ~3 router failures (have to immediately pull traffic for an hour) ~dozens of minor 30-second blips for dns ~1000 individual machine failures ~thousands of hard drive failures slow disks, bad memory, misconfigured machines, flaky machines, etc.

13.
File Systems OverviewSystem that permanently stores data

14.
Usually layered ontop of a lower-level physical storage medium

15.
Divided into logicalunits called “files” Addressable by a filename (“foo.txt”) Files are often organized into directories Usually supports hierarchical nesting (directories)

16.
A path isthe expression that joins directories and filename to form a unique “full name” for a file. Directories may further belong to a volume

17.
The set ofvalid paths form the namespace of the file system.

18.
What Gets StoredUser data itself is the bulk of the file system's contents

19.
Also includes meta-dataon a volume-wide and per-file basis: Volume-wide: Available space Formatting info character set ... Per-file: name owner modification date physical layout...

20.
High-Level Organization Filesare typically organized in a “tree” structure made of nested directories

21.
One directory actsas the “root”

22.
“links” (symlinks, shortcuts,etc) provide simple means of providing multiple access paths to one file

23.
Other file systemscan be “mounted” and dropped in as sub-hierarchies (other drives, network shares)

24.
Typical operations ona file: create, delete, rename, open, close, read, write, append. also lock for multi-user systems.

25.
Low-Level Organization (1/2)File data and meta-data stored separately

26.
File descriptors +meta-data stored in inodes (Un*x) Large tree or table at designated location on disk

27.
Tells how tolook up file contents Meta-data may be replicated to increase system reliability

28.
Low-Level Organization (2/2)“Standard” read-write medium is a hard drive (other media: CDROM, tape, ...)

29.
Viewed as asequential array of blocks

30.
Usually address ~1KB chunk at a time

31.
Tree structure is“flattened” into blocks

32.
Overlapping writes/deletes cancause fragmentation: files are often not stored with a linear layout inodes store all block numbers related to file

33.
Fragmentation

34.
Filesystem Design ConsiderationsNamespace: physical, logical

35.
Consistency: what todo when more than one user reads/writes on the same file?

36.
Security: who cando what to a file? Authentication/ACL

37.
Reliability: can filesnot be damaged at power outage or other hardware failures?

38.
Local Filesystems onUnix-like Systems Many different designs

39.
Namespace: root directory“/”, followed by directories and files.

40.
Consistency: “sequential consistency”,newly written data are immediately visible to open reads (if...)

41.
Security: uid/gid, modeof files

42.
kerberos: tickets Reliability:superblocks, journaling, snapshot more reliable filesystem on top of existing filesystem: RAID computer

43.
Namespace Physical mapping:a directory and all of its subdirectories are stored on the same physical media. /mnt/cdrom

44.
/mnt/disk1, /mnt/disk2, …when you have multiple disks Logical volume: a logical namespace that can contain multiple physical media or a partition of a physical media still mounted like /mnt/vol1

45.
dynamical resizing byadding/removing disks without reboot

46.
splitting/merging volumes aslong as no data spans the split

47.
Journaling Changes tothe filesystem is logged in a journal before it is committed. useful if an supposedly atomic action needs two or more writes, e.g., appending to a file (update metadata + allocate space + write the data).

48.
can play backa journal to recover data quickly in case of hardware failure. (old-fashioned “scan the volume” fsck anyone?) What to log? changes to file content: heavy overhead

49.
changes to metadata:fast, but data corruption may occur Implementations: xfs3, ReiserFS, IBM's JFS, etc.

50.
RAID Redundant Arrayof Inexpensive Disks

51.
Different RAID levelsRAID 0 (striped disks): data is distributed among disk for performance.

52.
RAID 1 (mirror):data is mirrored on another disk 1:1 for reliability.

53.
RAID 5 (stripeddisks with distributed parity): similar to RAID 0, but with a parity bit for data recovery in case one disk is lost. The parity bit is rotated among disks to maximize throughput.

54.
RAID 6 (stripeddisks with dual distributed parity): similar to RAID 5 but with 2 parity bits. Can lose 2 disks without losing data.

55.
RAID 10 (1+0,striped mirrors)

56.
Snapshot A snapshot= a copy of a set of files and directories at a point in time. read-only snapshots, read-write snapshots

57.
usually done bythe filesystem itself, sometimes by LVMs

58.
backing up datacan be done on a read-only snapshot without worrying about consistency Copy-on-write is a simple and fast way to create snapshots current data is the snapshot.

59.
a request towrite to a file creates a new copy, and work from there afterwards. Implementation: UFS, Sun's ZFS, etc.

60.
Should the filesystem be faster or more reliable?

61.
But faster atwhat: Large files? Small files? Lots of reading? Frequent writers, occasional readers?

62.
Block size Smallerblock size reduces amount of wasted space

63.
Larger block sizeincreases speed of sequential reads (may not help random access)

64.
Distributed File Systems

65.
Distributed Filesystems Supportaccess to files on remote servers

66.
Must support concurrencyMake varying guarantees about locking, who “wins” with concurrent writes, etc...

67.
Must gracefully handledropped connections Can offer support for replication and local caching

68.
Different implementations sitin different places on complexity/feature scale

69.
DFS is usefulwhen... Multiple users want to share files

70.
Users move fromone computer to another

71.
Users want auniform namespace for shared files on every computer

72.
Users want acentralized storage system – backup and management

73.
Note that a“user” of a DFS may actually be a “program”

74.
Features of DistributedFile Systems Different systems have different designs and behaviors on the following features. Interface: file system, block I/O, custom made

75.
Security: various authentication/authorizationschemes.

76.
Reliability (fault-tolerance): cancontinue to function when some hardware fail (disks, nodes, power, etc).

77.
Namespace (virtualization): canprovide logical namespace that can span across physical boundaries.

78.
Consistency: all clientsget the same data all the time. It is related to locking, caching, and synchronization.

79.
Parallel: multiple clientscan have access to multiple disks at the same time.

80.
Scope: local areanetwork vs. wide area network.

81.
NFS Firstdeveloped in 1980s by Sun

82.
Most widely knowndistributed filesystem.

83.
Presented with standardPOSIX FS interface

84.
Network drives aremounted into local directory hierarchy Your home directory at CINC is probably NFS-driven

85.
Type 'mount' sometime at the prompt if curious client client client server

86.
NFS Protocol Initiallycompletely stateless Operated over UDP; did not use TCP streams

87.
File locking, etc,implemented in higher-level protocols

88.
Implication: All clientrequests have enough info to complete op example: Client specifies offset in file to write to

89.
one advantage: Serverstate does not grow with more clients Modern implementations use TCP/IP & stateful protocols

90.
RFCs: RFC 1094for v2 (3/1989)

91.
RFC 1813 forv3 (6/1995)

92.
RFC 3530 forv4 (4/2003)

93.
NFS Overview RemoteProcedure Calls (RPC) for communication between client and server

94.
Client Implementation Providestransparent access to NFS file system UNIX contains Virtual File system layer (VFS)

95.
Vnode: interface forprocedures on an individual file Translates vnode operations to NFS RPCs Server Implementation Stateless: Must not have anything only in memory

96.
Implication: All modifieddata written to stable storage before return control to client Servers often add NVRAM to improve performance

97.
Server-side Implementation NFSdefines a virtual file system Does not actually manage local disk layout on server Server instantiates NFS volume on top of local file system Local hard drives managed by concrete file systems (ext*, ReiserFS, ...)

98.
Export local disksto clients

99.
NFS Locking NFSv4 supports stateful locking of files Clients inform server of intent to lock

100.
Server can notifyclients of outstanding lock requests

101.
Locking is lease-based:clients must continually renew locks before a timeout

102.
Loss of contactwith server abandons locks

103.
NFS Client CachingNFS Clients are allowed to cache copies of remote files for subsequent accesses

104.
Supports close-to-open cache consistency When client A closes a file, its contents are synchronized with the master, and timestamp is changed

105.
When client Bopens the file, it checks that local timestamp agrees with server timestamp. If not, it discards local copy.

106.
Concurrent reader/writers mustuse flags to disable caching

107.
Cache Consistency Problem:Consistency across multiple copies (server and multiple clients)

108.
How to keepdata consistent between client and server? If file is changed on server, will client see update?

109.
Determining factor: Readpolicy on clients How to keep data consistent across clients? If write file on client A and read on client B, will B see update?

110.
Determining factor: Writeand read policy on clients

111.
NFS: Summary Verypopular

112.
Full POSIX interfacemeans it “drops in” very easily.

113.
No cluster-wide uniformnamespace: each client decides how to name a volume mounted from an NFS server.

114.
No location transparency:data movement means remount

115.
NFS volume managedby single server Higher load on central server, bottleneck on scalability

116.
Simplifies coherency protocolsChallenges: Fault tolerance, scalable performance, and consistency

117.
AFS (The AndrewFile System) Developed at Carnegie Mellon University since 1983

118.
Strong security: Kerberosauthentication richer set of access control bits than UNIX: separate “administer”/“delete” bits, application-specific bits. Higher scalability compared with distributed file systems at the time. Supports 50,000+ clients at enterprise level.

119.
Global namespace: /afs/cmu.edu/vol1/...,WAN scope

120.
Location transparency: movingfiles just works.

121.
Heavily use clientside caching.

122.
Support read-only replication.client client client server replica

123.
Morgan Stanley's comparison:AFS vs. NFS Better client/server ratio NFS (circa 1993) topped out at 25:1

124.
AFS: 100s Robustvolume replication NFS servers go down, and take their clients with them

125.
AFS servers godown, nobody notices (OK, RO data only) WAN file sharing NFS couldn't do it reliably

126.
AFS worked likea charm Security was never an issue (kerberos4) http://www-conf.slac.stanford.edu/AFSBestPractices/Slides/MorganStanley.pdf

127.
Local Caching Filereads/writes operate on locally cached copy

128.
The modified localcopy is sent back to server when file is closed

129.
Open local copiesare notified of external updates through callbacks but not updated Trade-offs: Shared database files do not work well on this system

130.
Does not supportwrite-through to shared medium

131.
Replication AFS allowsread-only copies of filesystem volumes

132.
Copies are guaranteedto be atomic checkpoints of entire FS at time of read-only copy generation (“snapshot”)

133.
Modifying data requiresaccess to the sole r/w volume Changes do not propagate to existing read-only copies

134.
AFS: Summary Notquite POSIX Stronger security/permissions

135.
No file write-through High availability through replicas and local caching

136.
Scalability by reducingload on servers

137.
Not appropriate forall file types ACM Trans. on Computer Systems, 6(1):51-81, Feb 1988.

138.
Global File System

139.
Lustre Overview Developedby Peter Braam at Carnegie Mellon since 1999 then Cluster File System in 2001, acquired by Sun in 2007

140.
Goal: removing bottlenecksto achieve scalability Object-based: separate metadata and file data metadata server(s): MDS (namespace, ACL, attributes)

141.
object storage server(s):OSS

142.
client read/write directlywith OSS Consistency: Lustre distributed lock manager

143.
Performance: data canbe striped

144.
POSIX interface clientclient client OSS MDS OSS OSS MDS OSS OST

145.
Key Design Issue: Scalability I/O throughput How to avoid bottlenecks Metadata scalability How can 10,000’s of nodes work on files in same folder Cluster Recovery If sth fails, how can transparent recovery happen Management Adding, removing, replacing, systems; data migration & backup

146.
The Lustre Architecture

147.
Metadata Service (MDS)All access to the file is governed by MDS which will directly or indirectly authorize access.

148.
To control namespaceand manage inodes

149.
Load balanced clusterservice for the scalability (a well balanced API, a stackable framework for logical MDS, replicated MDS)

150.
Journaled batched metadataupdates

151.
Object Storage Targets(OST) Keep file data objects

152.
File I/O service:Access to the objects

153.
The block allocationfor data obj., leading distributed and scalability

154.
OST s/w modulesOBD server, Lock server

155.
Obj. storage driver,OBD filter

156.
Portal API

157.
Distributed Lock ManagerFor generic and rich lock service

158.
Lock resources: resourcedatabase Organize resources in trees High performance node that acquires resource manages tree Intent locking: utilization dependent locking modes write-back lock for low utilization files

159.
no lock forhigh contention files, write to OSS by DLM

160.
Metadata

161.
File I/O

162.
Striping

163.
Lustre Summary Goodscalability and high performance Lawrence Livermore National Lab BlueGene/L has 200,000 clients processes access 2.5PB Lustre fs through 1024 IO nodes over 1Gbit Ethernet network at 35 GB/sec. Reliability Journaled metadata in metadata cluster

164.
OSTs are responsiblefor their data Consistency

165.
Other file systemsworth mentioning Coda: post-AFS, pre-Lustre, from CMU

166.
Global File System:shared disk file systems

167.
PVFS by IBM

168.
ZFS by Sun

Distributed File System

More Related Content

What's hot

Similar to Distributed File System

Recently uploaded

Distributed File System

Editor's Notes