Memory Hierarchy Design and its Characteristics

In the Computer System Design, Memory Hierarchy is an enhancement to

organize the memory such that it can minimize the access time. The Memory
Hierarchy was developed based on a program behavior known as locality of
[Link] figure below clearly demonstrates the different levels of
memory
hierarchy :

This Memory Hierarchy Design is divided into 2 main types:

1. External Memory or Secondary Memory – Comprising of
Magnetic Disk, Optical Disk, Magnetic Tape i.e. peripheral storage
devices which are accessible by the processor via I/O Module.
2. Internal Memory or Primary Memory – Comprising of Main
Memory, Cache Memory & CPU registers. This is directly
accessible by the processor.
There are typically four levels of memory in a memory hierarchy:
Registers: Registers are small, high-speed memory units located in the
CPU. They are used to store the most frequently used data and instructions.
Registers have the fastest access time and the smallest storage capacity,
typically ranging from 16 to 64 bits.
Cache Memory: Cache memory is a small, fast memory unit located close to
the CPU. It stores frequently used data and instructions that have been
recently accessed from the main memory. Cache memory is designed to
minimize the time it takes to access data by providing the CPU with quick
access to frequently used data.
Main Memory: Main memory, also known as RAM (Random Access
Memory), is the primary memory of a computer system. It has a larger
storage capacity than cache memory, but it is slower. Main memory is used
to store data and instructions that are currently in use by the CPU.

Secondary Storage: Secondary storage, such as hard disk drives (HDD)

and solid-state drives (SSD), is a non-volatile memory unit that has a larger
storage capacity than main memory. It is used to store data and instructions
that are not currently in use by the CPU. Secondary storage has the slowest
access time and is typically the least expensive type of memory in the
memory hierarchy.
We can infer the following characteristics of Memory Hierarchy Design from
above figure:
1. Capacity: It is the global volume of information the memory can
store. As we move from top to bottom in the Hierarchy, the capacity
increases.
2. Access Time: It is the time interval between the read/write request
and the availability of the data. As we move from top to bottom in
the Hierarchy, the access time increases.
3. Performance: Earlier when the computer system was designed
without Memory Hierarchy design, the speed gap increases
between the CPU registers and Main Memory due to large
difference in access time. This results in lower performance of the
system and thus, enhancement was required. This enhancement
was made in the form of Memory Hierarchy Design because of
which the performance of the system increases. One of the most
significant ways to increase system performance is minimizing how
far down the memory hierarchy one has to go to manipulate data.
4. Cost per bit: As we move from bottom to top in the Hierarchy, the
cost per bit increases i.e., Internal Memory is costlier than External
Memory.
According to the memory Hierarchy, the system supported memory
standards are defined below:
Main Memory
The main memory acts as the central storage unit in a computer system. It is a
relatively large and fast memory which is used to store programs and data during the
run time operations.

The primary technology used for the main memory is based on semiconductor
integrated circuits. The integrated circuits for the main memory are classified into
two major units.

1. RAM (Random Access Memory) integrated circuit chips

2. ROM (Read Only Memory) integrated circuit chips

RAM integrated circuit chips

The RAM integrated circuit chips are further classified into two possible operating
modes, static and dynamic.
The primary compositions of a static RAM are flip-flops that store the binary information.
The nature of the stored information is volatile, i.e. it remains valid as long as power is
applied to the system. The static RAM is easy to use and takes less time performing read and
write operations as compared to dynamic RAM.

The dynamic RAM exhibits the binary information in the form of electric charges that
are applied to capacitors. The capacitors are integrated inside the chip by MOS
transistors. The dynamic RAM consumes less power and provides large storage
capacity in a single memory chip.

RAM chips are available in a variety of sizes and are used as per the system requirement.
The following block diagram demonstrates the chip interconnection in a 128 * 8 RAM
chip.

o A 128 * 8 RAM chip has a memory capacity of 128 words of eight bits (one byte)
per word. This requires a 7-bit address and an 8-bit bidirectional data bus.
o The 8-bit bidirectional data bus allows the transfer of data either from memory
to CPU during a read operation or from CPU to memory during
a write operation.
o The read and write inputs specify the memory operation, and the two chip
select (CS) control inputs are for enabling the chip only when the
microprocessor selects it.
o The bidirectional data bus is constructed using three-state buffers.
o The output generated by three-state buffers can be placed in one of the three
possible states which include a signal equivalent to logic 1, a signal equal to
logic 0, or a high-impedance state.
ROM integrated circuit
The primary component of the main memory is RAM integrated circuit chips, but a
portion of memory may be constructed with ROM chips.

A ROM memory is used for keeping programs and data that are permanently resident
in the computer.

Apart from the permanent storage of data, the ROM portion of main memory is needed for
storing an initial program called a bootstrap loader. The primary function of the bootstrap
loader program is to start the computer software operating when power is turned on.

ROM chips are also available in a variety of sizes and are also used as per the system
requirement. The following block diagram demonstrates the chip interconnection in a
512 * 8 ROM chip.

o A ROM chip has a similar organization as a RAM chip. However, a ROM can only
perform read operation; the data bus can only operate in an output mode.
o The 9-bit address lines in the ROM chip specify any one of the 512 bytes stored
in it.
o The value for chip select 1 and chip select 2 must be 1 and 0 for the unit to
operate. Otherwise, the data bus is said to be in a high-impedance state

Difference between SRAM and DRAM

Static Random Access Memory (SRAM): Data is stored in transistors and
requires a constant power flow. Because of the continuous power, SRAM doesn’t
need to be refreshed to remember the data being stored. SRAM is called static as
no change or action i.e. refreshing is not needed to keep the data intact. It is used
in cache memories.
Advantage: Low power consumption and faster access speeds.
Disadvantage: Fewer memory capacities and high costs of manufacturing.
Dynamic Random Access Memory (DRAM): Data is stored in capacitors.
Capacitors that store data in DRAM gradually discharge energy, no energy means
the data has been lost. So, a periodic refresh of power is required in order to
function. DRAM is called dynamic as constant change or action(change is
continuously happening) i.e. refreshing is needed to keep the data intact. It is used
to implement main memory.
Advantage: Low costs of manufacturing and greater memory capacities.
Disadvantage: Slow access speed and high power consumption .

Types of DRAM :
There are mainly 5 types of DRAM:
1. Asynchronous DRAM (ADRAM) –
The DRAM described above is the asynchronous type DRAM. The timing of the
memory device is controlled asynchronously. A specialized memory controller
circuit generates the necessary control signals to control the timing. The CPU
must take into account the delay in the response of the memory.

2. Synchronous DRAM (SDRAM) –

These RAM chips’ access speed is directly synchronized with the CPU’s clock.
For this, the memory chips remain ready for operation when the CPU expects
 them to be ready. These memories operate at the CPU-memory bus without
imposing wait states. SDRAM is commercially available as modules
incorporating multiple SDRAM chips and forming the required capacity for the
modules.

3. Double-Data-Rate SDRAM (DDR SDRAM) –

This faster version of SDRAM performs its operations on both edges of the
clock signal; whereas a standard SDRAM performs its operations on the rising
edge of the clock signal. Since they transfer data on both edges of the clock,
the data transfer rate is doubled. To access the data at high rate, the memory
cells are organized into two groups. Each group is accessed separately.

4. Rambus DRAM (RDRAM) –

The RDRAM provides a very high data transfer rate over a narrow CPU-
memory bus. It uses various speedup mechanisms, like synchronous memory
interface, caching inside the DRAM chips and very fast signal timing. The
Rambus data bus width is 8 or 9 bits.

5. Cache DRAM (CDRAM) –

This memory is a special type DRAM memory with an on-chip cache memory
(SRAM) that acts as a high-speed buffer for the main DRAM.
Types of Read-Only Memory (ROM)
1. PROM (Programmable Read-Only Memory)
2. EPROM (Erasable Programmable Read Only Memory)
3. EEPROM (Electrically Erasable Programmable Read Only Memory)
4. Flash memory
1. PROM (Programmable read-only memory): It can be programmed by the user. Once
programmed, the data and instructions in it cannot be changed.
2. EPROM (Erasable Programmable read-only memory): It can be reprogrammed. To
erase data from it, expose it to ultraviolet light. To reprogram it, erase all the previous data.
3. EEPROM (Electrically erasable programmable read-only memory): The data can be
erased by applying an electric field, with no need for ultraviolet light. We can erase only
portions of the chip.
4. Flash memory: A currently popular type of EEPROM, in which erasing is performed in
large blocks rather than bit by bit, is known as flash EPROM or flash memory. Erasing in
large blocks reduces the overhead circuitry, thus leading to greater density and lower cost.

CACHE MEMORY
Cache Memory is a special very high-speed memory. It is used to speed up and
synchronize with high-speed CPU. Cache memory is costlier than main memory or disk
memory but more economical than CPU registers. Cache memory is an extremely fast
memory type that acts as a buffer between RAM and the CPU. It holds frequently requested
data and instructions so that they are immediately available to the CPU when needed. Cache
memory is used to reduce the average time to access data from the Main memory. The
cache is a smaller and faster memory that stores copies of the data from frequently used
main memory locations. There are various different independent caches in a CPU, which
store instructions and
data.

Cache Performance: When the processor needs to read or write a location in main
memory, it first checks for a corresponding entry in the cache.
• If the processor finds that the memory location is in the cache, a cache hit has
occurred and data is read from the cache.
• If the processor does not find the memory location in the cache, a cache
miss has occurred. For a cache miss, the cache allocates a new entry and copies
in data from main memory, then the request is fulfilled from the contents of the
cache.
The performance of cache memory is frequently measured in terms of a quantity called Hit
ratio.
Hit ratio(H) = hit / (hit + miss) = no. of hits/total accesses
Miss ratio = miss / (hit + miss) = no. of miss/total accesses = 1 - hit ratio(H)
We can improve Cache performance using higher cache block size, and higher
associativity, reduce miss rate, reduce miss penalty, and reduce the time to hit in the
cache.
Cache Mapping: There are three different types of mapping used for the purpose of cache
memory which is as follows: Direct mapping, Associative mapping, and Set-Associative
mapping. These are explained below.
A. Direct Mapping
The simplest technique, known as direct mapping, maps each block of main memory into
only one possible cache line. or In Direct mapping, assign each memory block to a specific
line in the cache. If a line is previously taken up by a memory block when a new block
needs to be loaded, the old block is trashed. An address space is split into two parts index
field and a tag field. The cache is used to store the tag field whereas the rest is stored in
the main memory. Direct mapping`s performance is directly proportional to the Hit ratio.
1. For purposes of cache access, each main memory address can be viewed as
consisting of three fields. The least significant w bits identify a unique word or
byte within a block of main memory. In most contemporary machines, the
address is at the byte level. The remaining s bits specify one of the 2 s blocks of
main memory. The cache logic interprets these s bits as a tag of s-r bits (most
significant portion) and a line field of r bits. This latter field identifies one of the
m=2r lines of the cache. Line offset is index bits in the direct mapping.
B. Associative Mapping
In this type of mapping, the associative memory is used to store content and addresses of
the memory word. Any block can go into any line of the cache. This means that the word id
bits are used to identify which word in the block is needed, but the tag becomes all of the
remaining bits. This enables the placement of any word at any place in the cache memory.
It is considered to be the fastest and the most flexible mapping form. In associative
mapping the index bits are zero.

C. Set-associative Mapping
This form of mapping is an enhanced form of direct mapping where the drawbacks of
direct mapping are removed. Set associative addresses the problem of possible thrashing
in the direct mapping method. It does this by saying that instead of having exactly one line
that a block can map to in the cache, we will group a few lines together creating a set.
Then a block in memory can map to any one of the lines of a specific set. Set-associative
mapping allows that each word that is present in the cache can have two or more words in
the main memory for the same index address. Set associative cache mapping combines
the best of direct and associative cache mapping techniques. In set associative mapping
the index bits are given by the set offset bits. In this case, the cache consists of a number
of sets, each of which consists of a number of lines. The relationships are

A few instructions in the localized areas in memory are executed repeatedly for some time duration and
other instructions are accessed infrequently. This phenomenon is known as the property of locality of
reference.
There are two dimensions of the locality of reference property: temporal and spatial.
1. Spatial Locality of reference This says that there is a chance that the element
will be present in close proximity to the reference point and next time if again
searched then more close proximity to the point of reference.
 2. Temporal Locality of reference In this Least recently used algorithm will be
used. Whenever there is page fault occurs within a word will not only load word
in main memory but complete page fault will be loaded because the spatial
locality of reference rule says that if you are referring to any word next word will
be referred to in its register that’s why we load complete page table so the
complete block will be loaded.

Parts of programs and data are brought into main memory from secondary memory, as the CPU needs
them. Virtual memory is a technique used in some large computer systems, which gives the programmer
an illusion of having a large main memory, although which may not be the case. The size of
virtual memory is equivalent to the size of secondary memory. Each address referenced by the CPU
called the virtual (logical) address is mapped to a physical address in main memory. This mapping is
done during run-time and is performed by a hardware device called memory-management unit (MMU)
with the help of a memory map table, which is maintained by the operating system.
The virtual memory makes the task of programming much easier, because the programmer no
longer needs to bother about the amount of physical memory available. For example, a program size
is 18 MB and the available user part of the main memory is 15 MB (Other part of the main memory is
occupied by the operating system). First, 15 MB of the program is loaded into main memory and then
remaining 3 MB is still in the secondary memory. When the remaining 3 MB code is needed for
execution, swap out the 3 MB code from main memory to secondary memory and swap in new 3 MB
code from secondary memory to main memory.
The advantage of virtual memory is efficient utilization of main memory, because the larger size
program is divided into blocks and partially each block is loaded in the main memory whenever it is
required. Thus, multiple programs can be executed simultaneously. The technique of virtual memory
has other advantages of efficient CPU utilization and improved throughput.

Logical (Virtual) Address Space and Physical Address Space When a program needs to be
executed, the CPU would generate addresses called logical addresses. The corresponding addresses in
the physical memory, as occupied by the executing program, are called physical addresses. The set of
all logical addresses generated by the CPU or program is called logical-address space and the set of
all physical addresses corresponding to these logical addresses is called physical-address space. The
memory-management unit (MMU) maps each logical address to a physical address during program
execution. The Fig. 4.30 illustrates this mapping method, which uses a special register called base
register or relocation register. The content of the relocation register is added to every logical address
generated by the user program at the beginning of execution. For example, if the relocation register
holds an address value 2000, then a reference to the location 0 by the user is dynamically relocated to
2000 address. A reference to the address 150 is mapped to the address 2150.

A virtual memory system may be configured in one of the following ways:

1. Paging technique
2. Segmentation technique.

Paging
Paging is a memory management scheme that eliminates the need for contiguous
allocation of physical memory. The process of retrieving processes in the form of pages
from the secondary storage into the main memory is known as paging. The basic purpose
of paging is to separate each procedure into pages. Additionally, frames will be used to
split the main memory. This scheme permits the physical address space of a process to be
non – contiguous.
Features of paging:
1. Mapping logical address to physical address.
2. Page size is equal to frame size.
3. Number of entries in a page table is equal to number of pages in logical
address space.
4. The page table entry contains the frame number.
5. All the page table of the processes are placed in main memory.
Example:
• If Logical Address = 31 bit, then Logical Address Space = 231 words = 2 G
words (1 G = 230)
• If Logical Address Space = 128 M words = 27 * 220 words, then Logical Address
= log2 227 = 27 bits
• If Physical Address = 22 bit, then Physical Address Space = 222 words = 4 M
words (1 M = 220)
• If Physical Address Space = 16 M words = 24 * 220 words, then Physical
Address = log2 224 = 24 bits
The mapping from virtual to physical address is done by the memory management unit
(MMU) which is a hardware device and this mapping is known as paging technique.
• The Physical Address Space is conceptually divided into a number of fixed-size
blocks, called frames.
• The Logical address Space is also splitted into fixed-size blocks, called pages.
• Page Size = Frame Size
Let us consider an example:
• Physical Address = 12 bits, then Physical Address Space = 4 K words
• Logical Address = 13 bits, then Logical Address Space = 8 K words
• Page size = frame size = 1 K words (assumption)
 Address generated by CPU is divided into
• Page number(p): Number of bits required to represent the pages in Logical
Address Space or Page number
• Page offset(d): Number of bits required to represent particular word in a page
or page size of Logical Address Space or word number of a page or page
offset.
Physical Address is divided into
• Frame number(f): Number of bits required to represent the frame of Physical
Address Space or Frame number.
• Frame offset(d): Number of bits required to represent particular word in a
frame or frame size of Physical Address Space or word number of a frame or
frame offset.
The hardware implementation of page table can be done by using dedicated registers.
But the usage of register for the page table is satisfactory only if page table is small. If
page table contain large number of entries then we can use TLB(translation Look-aside
buffer), a special, small, fast look up hardware cache.
• The TLB is associative, high speed memory.
• Each entry in TLB consists of two parts: a tag and a value.
• When this memory is used, then an item is compared with all tags
simultaneously. If the item is found, then corresponding value is returned.
Paging is a memory management technique used in operating systems to manage
memory and allocate memory to processes. In paging, memory is divided into fixed-size
blocks called pages, and processes are allocated memory in terms of these pages. Each
page is of the same size, and the size is typically a power of 2, such as 4KB or 8KB.
In paging, the physical memory is divided into fixed-size blocks called page frames, which
are the same size as the pages used by the process. The process’s logical address space
is also divided into fixed-size blocks called pages, which are the same size as the page
frames. When a process requests memory, the operating system allocates one or more
page frames to the process and maps the process’s logical pages to the physical page
frames.
The mapping between logical pages and physical page frames is maintained by the page
table, which is used by the memory management unit to translate logical addresses into
physical addresses. The page table maps each logical page number to a physical page
frame number.

Here are some important points about paging in operating systems:

1. Paging reduces internal fragmentation since memory is allocated in fixed-size

blocks.
2. Paging allows for more efficient use of memory since it enables memory to be
allocated on demand.
3. Paging allows for protection and sharing of memory between processes.
4. Paging can lead to external fragmentation, where memory is fragmented into
small, non-contiguous blocks, making it difficult to allocate large blocks of
memory to a process.
5. Paging involves overhead due to the maintenance of the page table and the
translation of logical addresses to physical addresses
Segmentation
A process is divided into Segments. The chunks that a program is divided into which are
not necessarily all of the same sizes are called segments. Segmentation gives the user’s
view of the process which paging does not give.
There is no simple relationship between logical addresses and physical addresses in
segmentation. A table stores the information about all such segments and is called
Segment Table.
Segment Table – It maps two-dimensional Logical address into one-dimensional Physical
address. It’s each table entry has:
• Base Address: It contains the starting physical address where the segments
reside in memory.
• Limit: It specifies the length of the segment.
Translation of Two dimensional Logical Address to dimensional Physical
Address.

Address generated by the CPU is divided into:

 • Segment number (s): Number of bits required to represent the segment.
• Segment offset (d): Number of bits required to represent the size of the segment.
Advantages of Segmentation –
• No Internal fragmentation. Segment Table consumes less space in comparison to Page
table in paging.
• As a complete module is loaded all at once, segmentation improves CPU utilization.
• The user’s perception of physical memory is quite similar to segmentation. Users can
divide user programs into modules via segmentation. These modules are nothing more
than the separate processes’ codes.
• The user specifies the segment size, whereas in paging, the hardware determines the
page size.
• Segmentation is a method that can be used to segregate data from security operations.
• Flexibility: Segmentation provides a higher degree of flexibility than paging. Segments
can be of variable size, and processes can be designed to have multiple segments,
allowing for more fine-grained memory allocation.
• Sharing: Segmentation allows for sharing of memory segments between processes.
This can be useful for inter-process communication or for sharing code libraries.
• Protection: Segmentation provides a level of protection between segments, preventing
one process from accessing or modifying another process’s memory segment. This can
help increase the security and stability of the system.
Disadvantage of Segmentation –
• As processes are loaded and removed from the memory, the free memory space is
broken into little pieces, causing External fragmentation.
• Overhead is associated with keeping a segment table for each activity.
• Due to the need for two memory accesses, one for the segment table and the other for
main memory, access time to retrieve the instruction increases.
• Fragmentation: As mentioned, segmentation can lead to external fragmentation as
memory becomes divided into smaller segments. This can lead to wasted memory and
decreased performance.
• Overhead: The use of a segment table can increase overhead and reduce
performance. Each segment table entry requires additional memory, and accessing the
table to retrieve memory locations can increase the time needed for memory
operations.
• Complexity: Segmentation can be more complex to implement and manage than
paging. In particular, managing multiple segments per process can be challenging, and
the potential for segmentation faults can increase as a result.
Page Replacement
In an operating system that uses paging for memory management, a page replacement
algorithm is needed to decide which page needs to be replaced when a new page comes
in.
Page Fault: A page fault happens when a running program accesses a memory page that
is mapped into the virtual address space but not loaded in physical memory. Since actual
physical memory is much smaller than virtual memory, page faults happen. In case of a
page fault, Operating System might have to replace one of the existing pages with the
newly needed page. Different page replacement algorithms suggest different ways to
decide which page to replace. The target for all algorithms is to reduce the number of page
faults.

Page Replacement Algorithms:

1. First In First Out (FIFO): This is the simplest page replacement algorithm. In this
algorithm, the operating system keeps track of all pages in the memory in a queue, the
oldest page is in the front of the queue. When a page needs to be replaced page in the
front of the queue is selected for removal.
Example 1: Consider page reference string 1, 3, 0, 3, 5, 6, 3 with 3 page [Link] the
number of page faults.

Initially, all slots are empty, so when 1, 3, 0 came they are allocated to the empty slots —
> 3 Page Faults.
when 3 comes, it is already in memory so —> 0 Page Faults. Then 5 comes, it is not
available in memory so it replaces the oldest page slot i.e 1. —>1 Page Fault. 6 comes, it
is also not available in memory so it replaces the oldest page slot i.e 3 —>1 Page
Fault. Finally, when 3 come it is not available so it replaces 0 1 page fault.
Belady’s anomaly proves that it is possible to have more page faults when increasing the
number of page frames while using the First in First Out (FIFO) page replacement
algorithm. For example, if we consider reference strings 3, 2, 1, 0, 3, 2, 4, 3, 2, 1, 0, 4,
and 3 slots, we get 9 total page faults, but if we increase slots to 4, we get 10-page faults.
2. Optimal Page replacement: In this algorithm, pages are replaced which would not be
used for the longest duration of time in the future.
Example-2: Consider the page references 7, 0, 1, 2, 0, 3, 0, 4, 2, 3, 0, 3, 2, 3 with 4 page
frame. Find number of page fault.
Initially, all slots are empty, so when 7 0 1 2 are allocated to the empty slots —> 4 Page
faults
0 is already there so —> 0 Page fault. when 3 came it will take the place of 7 because it is
not used for the longest duration of time in the future.—>1 Page fault. 0 is already there
so —> 0 Page fault. 4 will takes place of 1 —> 1 Page Fault.
Now for the further page reference string —> 0 Page fault because they are already
available in the memory.
Optimal page replacement is perfect, but not possible in practice as the operating system
cannot know future requests. The use of Optimal Page replacement is to set up a
benchmark so that other replacement algorithms can be analyzed against it.
3. Least Recently Used: In this algorithm, page will be replaced which is least recently
used.
Example-3: Consider the page reference string 7, 0, 1, 2, 0, 3, 0, 4, 2, 3, 0, 3, 2, 3 with 4
page frames. Find number of page faults.

Initially, all slots are empty, so when 7 0 1 2 are allocated to the empty slots —> 4 Page
faults
0 is already their so —> 0 Page fault. when 3 came it will take the place of 7 because it is
least recently used —>1 Page fault
0 is already in memory so —> 0 Page fault.
4 will takes place of 1 —> 1 Page Fault
Now for the further page reference string —> 0 Page fault because they are already
available in the memory.