File System Implementation - Part1

File System Implementation (Part I)
Amir H. Payberah
amir@sics.se
Amirkabir University of Technology
(Tehran Polytechnic)
Amir H. Payberah (Tehran Polytechnic) File System Implementation 1393/9/8 1 / 57

Motivation

Motivation
The ﬁle system resides permanently on secondary storage.
How to
• structure ﬁle use
• allocate disk space
• recover free space
• track the locations of data
• interface other parts of the OS to secondary storage

File System Structure

File-System Structure
Disk provides in-place rewrite and random access
• I/O transfers performed in blocks of sectors (usually 512 bytes)

File system resides on secondary storage
• User interface to storage, mapping logical to physical
• Eﬃcient and convenient access to disk

File system resides on secondary storage
• User interface to storage, mapping logical to physical
• Eﬃcient and convenient access to disk
File structure
• Logical storage unit
• Collection of related information

File-System Design Problems
How the ﬁle system should look to the user?

• Defining a file and its attributes
• The operations allowed on a file
• The directory structure for organizing files

• Defining a file and its attributes
• The operations allowed on a file
• The directory structure for organizing files
Algorithms and data structures to map the logical file system onto
the physical secondary-storage devices.

File System Layers (1/6)
Diﬀerent levels
Each level uses the features of lower
levels to create new features for use
by higher levels.
Reducing complexity and redundancy,
but adds overhead and can decrease
performance.

Device drivers manage I/O devices
at the I/O control layer.
Translates high-level commands to
low-level hardware-speciﬁc instructions.

Basic ﬁle system translates given command
like retrieve block 123 to device driver.

Basic file system translates given command
like retrieve block 123 to device driver.
Also manages memory buffers and
caches (allocation, freeing, replacement)
• Buffers hold data in transit
• Caches hold frequently used data

File organization understands ﬁles, logical
address, and physical blocks.

Translates logical block number to physical
block number.

Translates logical block number to physical
block number.
Manages free space and disk allocation.

Logical ﬁle system manages metadata
information.

information.
Translates file name into file number, file
handle, location by maintaining file
control blocks (inodes in Unix)

information.
Translates file name into file number, file
handle, location by maintaining file
control blocks (inodes in Unix)
Directory management
Protection

Many ﬁle systems, sometimes many within an OS
Each with its own format
• CD-ROM: ISO 9660
• Unix: UFS, FFS
• Windows: FAT, FAT32, NTFS
• Linux: more than 40 types, with extended ﬁle system (ext2, ext3,
ext4)

File System Implementation

File-System Implementation
Based on several on-disk and in-memory structures.

On-disk
• Boot control block (per volume)
• Volume control block (per volume)
• Directory structure (per ﬁle system)
• File control block (per ﬁle)

On-disk
• Boot control block (per volume)
• Volume control block (per volume)
• Directory structure (per file system)
• File control block (per file)
In-memory
• Mount table
• Directory structure cache
• The open-file table (system-wide and per process)
• Buffers of the file-system blocks

On-Disk File System Structures (1/2)
Boot control block contains information needed by system to boot
OS from that volume.

• Needed if volume contains OS, usually ﬁrst block of volume.
• In UFS, it is called boot block, and in NTFS partition boot sector.

Volume control block contains volume details.

Volume control block contains volume details.
• Total num. of blocks, num. of free blocks, block size, free block
pointers or array
• In UFS, it is called super block, and in NTFS master ﬁle table.

Directory structure organizes the files.
• In UFS, this includes file names and associated inode numbers.
• In NTFS, it is stored in the master file table.

Directory structure organizes the files.
• In UFS, this includes file names and associated inode numbers.
• In NTFS, it is stored in the master file table.
File control block contains many details about the file.
• In UFS, inode number, permissions, size, dates.
• In NFTS stores into in master file table.

In-Memory File System Structures
Mount table contains information about each mounted volume.

Directory structure cache holds the directory information of recently
accessed directories.

System-wide open-ﬁle table contains a copy of the FCB of each open
ﬁle.

file.
Per-process open-file table contains a pointer to the appropriate
entry in the system-wide open-file table.

file.
Per-process open-file table contains a pointer to the appropriate
entry in the system-wide open-file table.
Buffers hold file-system blocks when they are being read from disk
or written to disk.

Create a File
A program calls the logical ﬁle system.

Create a File
The logical ﬁle system knows the format of the directory structures,
and allocates a new FCB.

Create a File
The logical ﬁle system knows the format of the directory structures,
and allocates a new FCB.
The system, then, reads the appropriate directory into memory, up-
dates it with the new ﬁle name and FCB, and writes it back to the
disk.

Open a File
The file must be opened.
• The open() passes a file name to the logical file system.

Open a File
The open() first searches the system-wide open-file: if the file is
already in use by another process.

Open a File
• If yes: a per-process open-ﬁle table entry is created.

Open a File
• If no: the directory structure is searched for the given file name:
once the file is found, the FCB is copied into a system-wide
open-file table in memory.

Open a File
• If no: the directory structure is searched for the given file name:
once the file is found, the FCB is copied into a system-wide
open-file table in memory.
This table stores the FCB as well as the number of processes that
have the file open.

Read From a File
The open() returns a pointer to the appropriate entry in the per-
process file-system table.
All file operations are then performed via this pointer.
This pointer is called file descriptor in Unix and file handle in Win-
dows.

Close a File
When a process closes the ﬁle:
• The per-process table entry is removed.
• The system-wide entry’s open count is decremented.

Close a File
When a process closes the file:
• The per-process table entry is removed.
• The system-wide entry’s open count is decremented.
When all users that have opened the file close it, any updated meta-
data is copied back to the disk-based directory structure, and the
system-wide open-file table entry is removed.

Partitions and Mounting (1/2)
Partition can be a volume containing a ﬁle system or raw.
• Raw partition: just a sequence of blocks with no ﬁle system.

Partition can be a volume containing a file system or raw.
• Raw partition: just a sequence of blocks with no file system.
Boot block points to boot volume or boot loader.
• Boot loader: knows enough about the file-system structure to be
able to find and load the kernel and start it executing.
• Dual-boot that allows to install multiple OS on a single system.

Root partition contains the OS
• Mounted at boot time
• Other partitions can hold other OSes, other ﬁle systems, or be raw
• Other partitions can mount automatically or manually

Root partition contains the OS
• Mounted at boot time
• Other partitions can hold other OSes, other file systems, or be raw
• Other partitions can mount automatically or manually
At mount time, file system consistency checked.
• Is all metadata correct? if not, fix it, try again, if yes, add to mount
table, allow access

Virtual File Systems

Virtual File Systems (1/2)
Virtual File Systems (VFS) on Unix provide an object-oriented way
of implementing ﬁle systems.

VFS allows the same system call interface (the API) to be used for
diﬀerent types of ﬁle systems.

VFS allows the same system call interface (the API) to be used for
different types of file systems.
The API is to the VFS interface, rather than any specific type of
file system.

VFS layer serves two important functions:

1 It separates file-system-generic operations from their implementa-
tion, and allows transparent access to different types of file systems
mounted locally.

mounted locally.
2 It provides a mechanism for uniquely representing a ﬁle throughout
a network.

mounted locally.
a network.
The VFS is based on a structure, called a vnode.

mounted locally.
a network.
• Contains a numerical designator for a network-wide unique ﬁle.

mounted locally.
a network.
• Unix inodes are unique within only a single ﬁle system.

mounted locally.
a network.
• Unix inodes are unique within only a single ﬁle system.
• The kernel maintains one vnode structure for each active node.

VFS in Linux (1/2)
The four main object types deﬁned by the Linux VFS are:

VFS in Linux (1/2)
• The inode object: represents an individual ﬁle

VFS in Linux (1/2)
• The ﬁle object: represents an open ﬁle

VFS in Linux (1/2)
• The super block object: represents an entire ﬁle system

VFS in Linux (1/2)
• The super block object: represents an entire ﬁle system
• The dentry object: represents an individual directory entry

VFS in Linux (2/2)
VFS deﬁnes a set of operations on the objects that must be imple-
mented.

VFS in Linux (2/2)
mented.
Every object has a pointer to a function table.

VFS in Linux (2/2)
mented.
• Function table has addresses of routines to implement that function
on that object.

VFS in Linux (2/2)
mented.
• Function table has addresses of routines to implement that function
on that object.
• For example:
int open(...): open a file
int close(...): close an already-open file
ssize t read(...): read from a file
ssize t write(...): write to a file
int mmap(...): memory-map a file

Directory Implementation

Directory Implementation
Linear list
Hash table

Directory Implementation - Linear List
Linear list of ﬁle names with pointer to the data blocks.

Simple to program.

Simple to program.
Time-consuming to execute.

Simple to program.
Linear search time.

Simple to program.
Linear search time.
Could keep ordered alphabetically via linked list or use B+ tree:
binary search, but heavy

Directory Implementation - Hash Table
Hash Table: linear list with hash data structure

Decreases directory search time

Collisions: situations where two ﬁle names hash to the same location

Collisions: situations where two ﬁle names hash to the same location
Chained-overﬂow method.
• Each hash entry can be a linked list instead of an individual value.

Allocation Methods

Allocation Methods
How disk blocks are allocated to ﬁles?

Allocation Methods
How disk blocks are allocated to ﬁles?
Methods:
• Contiguous allocation
• Linked allocation
• Indexed allocation

Contiguous Allocation

Contiguous Allocation (1/2)
Contiguous allocation: each ﬁle occupies set of contiguous blocks.

• Best performance in most cases
• Simple: only starting location (block number) and length (number
of blocks) are required.

• Supports both sequential and direct access.

• Supports both sequential and direct access.
Allocation strategies like contiguous memory allocation:
• First fit
• Best fit
• Worst fit

Contiguous Allocation Problems
Finding space for ﬁle

External fragmentation

Need for compaction (fragmentation) oﬀ-line or on-line: lose of
performance

Need for compaction (fragmentation) oﬀ-line or on-line: lose of
performance
Knowing ﬁle size

Extent-Based Systems
A modiﬁed contiguous allocation scheme.
• E.g., Veritas ﬁle system

Extent-based ﬁle systems allocate disk blocks in extents.

Extent-based file systems allocate disk blocks in extents.
An extent is a contiguous block of disks.
• Extents are allocated for file allocation.
• A file consists of one or more extents.

Linked Allocation

Linked Allocation (1/2)
Linked allocation: each ﬁle is a linked list of blocks.
• Each block contains pointer to next block.
• File ends at null pointer.

No external fragmentation, no compaction.

No external fragmentation, no compaction.
Free space management system called when new block needed.

Linked Allocation Problems
Locating a block can take many I/Os and disk seeks.

Reliability can be a problem.

Reliability can be a problem.
The space required for the pointers.
• Eﬃciency can be improved by clustering blocks into groups but
increases internal fragmentation.

File-Allocation Table (FAT)
Beginning of volume has a table, indexed by block number.
Much like a linked list, but faster on disk and cacheable.

Indexed Allocation

Indexed Allocation (1/2)
Indexed allocation: each ﬁle has its own index block(s) of pointers
to its data blocks.

to its data blocks.
Need index table

to its data blocks.
Need index table
Random access

to its data blocks.
Need index table
Random access
Dynamic access without external fragmentation, but have overhead
of index block

Indexed Allocation Problems
Wasted space: overhead of the index blocks.
For example, even with a ﬁle of only one or two blocks, we need an
an entire index block.

Index Block Size
How large the index block should be?

Index Block Size
Keep the index block as small as possible.
• We need a mechanism to hold pointers for large ﬁles.

Index Block Size
Keep the index block as small as possible.
• We need a mechanism to hold pointers for large ﬁles.
Mechanisms for this purpose include the following:
• Linked scheme
• Multi-level index
• Combined scheme

Linked Scheme
Linked scheme: link blocks of index table (no limit on size)

Linked Scheme
Linked scheme: link blocks of index table (no limit on size)
For example, an index block might contain a small header giving the
name of the ﬁle and a set of the ﬁrst 100 disk-block addresses.
The next address is null or is a pointer to another index block.

Multi-Level Index
Two-level index
A ﬁrst-level index block to point to a set of second-level index blocks,
which in turn point to the ﬁle blocks.
Could be continued to a third or fourth level.

Combined Scheme
Combine scheme: used in Unix/Linux FS
The ﬁrst 12 pointers point to direct blocks
• The data for small ﬁles do not need a separate index block.
The next 3 pointers point to indirect blocks.
• Single indirect
• Double indirect
• Triple indirect

Performance
Best method depends on ﬁle access type.

Performance
Contiguous is great for sequential and random.

Performance
Linked is good for sequential, not random.

Performance
Linked is good for sequential, not random.
Indexed is more complex
• Single block access could require 2 index block reads then data
block read
• Clustering can help improve throughput, reduce CPU overhead

Summary

Summary
FS layers: device, I/O control, basic FS, ﬁle-organization, logical
FS, application

Summary
FS, application
FS implementation:
• On-disk structures: boot control block, volume control block,
directory structure, and file control block
• In-memory structures: mount table, directory structure, open-file
tables, and buffers

Summary
FS, application
FS implementation:
Virtual ﬁle system (VFS)

Summary
FS, application
FS implementation:
Directory implementation: linear list, and hash table

Summary
FS, application
FS implementation:
Directory implementation: linear list, and hash table
Allocation methods: contiguous allocation, linked allocation, and
indexed allocation

Questions?
Acknowledgements
Some slides were derived from Avi Silberschatz slides.

File System Implementation - Part1

More Related Content

What's hot(20)

Similar to File System Implementation - Part1(20)

More from Amir Payberah(20)

Recently uploaded(20)

File System Implementation - Part1