Advanced Computer
Architectures
Introduction to
Multiprocessors
1
Multitasking vs Multithreading vs Multiprocessing
1.Multiprogramming – Multiprogramming is known as keeping multiple
programs in the main memory at the same time ready for execution.
2.Multiprocessing – A computer using more than one CPU at a time.
3.Multitasking – Multitasking is nothing but multiprogramming with a Round-
robin scheduling algorithm.
4.Multithreading is an extension of multitasking.
2
Introduction
● Initial computer performance improvements
came from use of:
– Innovative manufacturing techniques.
● In later years,
– Most improvements came from exploitation of ILP.
– Both software and hardware techniques are being
used.
– Pipelining, dynamic instruction scheduling, out of
order execution, VLIW, vector processing, etc.
● ILP now appears fully exploited:
– Further performance improvements from ILP
appears limited.
3
Thread and Process-Level
Parallelism
● The way to achieve higher performance:
– Of late, exploitation of thread and process-
level parallelism is being focused.
● Exploit parallelism existing across
multiple processes or threads:
– Cannot be exploited by any ILP processor.
● Consider a banking application:
– Individual transactions can be executed in
parallel.
4
Processes versus Threads
● Processes:
– A process is a program in execution.
– An application normally consists of
multiple processes.
● Threads:
– A process consists of one of more
threads.
– Threads belonging to the same process
share data, and code space.
5
Single and Multithreaded
Processes
6
How can Threads be Created?
● By using any of the popular
thread libraries:
– POSIX Pthreads
– Win32 threads
– Java threads, etc.
7
User Threads
● Thread management done in user
space.
● User threads are supported and
managed without kernel support.
– Invisibleto the kernel.
– If one thread blocks, entire
process blocks.
– Limited benefits of threading.
8
Kernel Threads
● Kernel threads supported and
managed directly by the OS.
– Kernel creates Light Weight Processes
(LWPs).
● Most modern OS support kernel
threads:
– Windows XP/2000
– Solaris
– Linux
– Mac OS, etc.
9
Benefits of Threading
● Responsiveness:
– Threads share code, and data.
– Thread creation and switching
therefore much more efficient than
that for processes;
● As an example in Solaris:
– Creating threads 30x less costly
than processes.
– Context switching about 5x faster
than processes. 10
Benefits of Threading
cont…
● Truly concurrent execution:
– Possible
with processors
supporting concurrent execution
of threads: SMP, multi-core,
SMT (hyper threading), etc.
11
A Few Thread Examples
● Independent threads occur
naturally in several applications:
– Web server: different http
requests are the threads.
– File server
– Name server
– Banking: independent transactions
– Desktop applications: file loading,
display, computations, etc. can be
threads. 12
Reflection on Threading
● To think of it:
– Threadingis inherent to any
server application.
● Threads are also easily
identifiable in traditional
applications:
– Banking, Scientific computations,
etc.
13
Thread-level Parallelism
--- Cons cont…
● Threads with severe
dependencies:
– Maymake multithreading an
exercise in futility.
● Also not as “programmer
friendly” as ILP.
14
Thread Vs. Process-Level
Parallelism
● Threads are light weight (or fine-
grained):
– Threads share address space, data, files etc.
– Even when extent of data sharing and
synchronization is low: Exploitation of
thread-level parallelism meaningful only when
communication latency is low.
– Consequently, shared memory architectures
(UMA) are a popular way to exploit thread-
level parallelism.
15
A Broad Classification of
Computers
● Shared-memory multiprocessors
– Also called UMA
● Distributed memory computers
– Also called NUMA:
● Distributed Shared-memory (DSM)
architectures
● Clusters
● Grids, etc.
16
UMA vs. NUMA Computers
Latency = several
milliseconds to seconds
P1 P2 Pn P1 P2 Pn
Cache Cache Cache Cache Cache Cache
Bus
Main Main Main
Memory Memory Memory
Main
Memory
Network
Latency = 100s of ns
(a) UMA Model (b) NUMA Model
17
Distributed Memory
Computers
● Distributed memory computers use:
– Message Passing Model
● Explicit message send and receive
instructions have to be written by the
programmer.
– Send: specifies local buffer + receiving
process (id) on remote computer (address).
– Receive: specifies sending process on
remote computer + local buffer to place
data. 18
Advantages of Message-
Passing Communication
● Hardware for communication and
synchronization are much simpler:
– Compared to communication in a shared memory
model.
● Explicit communication:
– Programs simpler to understand, helps to reduce
maintenance and development costs.
● Synchronization is implicit:
– Naturally associated with sending/receiving
messages.
– Easier to debug. 19
Disadvantages of Message-Passing
Communication
● Programmer has to write explicit
message passing constructs.
– Also,
precisely identify the
processes (or threads) with which
communication is to occur.
● Explicit calls to operating
system:
– Higher overhead. 20
