Introduction to

Parallel Processing
C H A PT ER 2
Parallel Computing
• Parallel computing is a computing where the jobs are broken into discrete
parts that can be executed concurrently. 
• Each part is further broken down to a series of instructions. Instructions
from each part execute simultaneously on different CPUs.
• Parallel systems deal with the simultaneous use of multiple computer
resources that can include a single computer with multiple processors, a
number of computers connected by a network to form a parallel processing
cluster or a combination of both.
• Parallel systems are more difficult to program than computers with a single
processor because the architecture of parallel computers varies
accordingly and the processes of multiple CPUs must be coordinated and
synchronized.
The State of Computing
• Modern computers are equipped with powerful hardware facilities
driven by extensive software packages.
1. To assess state-of-the-art computing, we first review historical
milestones in the development of computers.
2. Then we take a grand tour of the crucial hardware and software
elements built into modern computer systems.
1. Computer Development Milestones
• Prior to I945, computers were made with mechanical or electromechanical
parts. The earliest mechanical computer can be traced back to 5110 BC in the
form of the abacus used in China.
• Blaise Pascal built a mechanical adder/ subtractor in France in 1642.
• Charles Babbage designed a difference engine in England for polynomial
evaluation in 1827.
• Konrad Zusc built the first binary mechanical computer in Germany in 1941.
• Howard Aiken proposed the very first electromechanical decimal computer,
which was built as the Harvard Mark I by IBM in 1944. Both Zusc’s and
Aiken’s machines were designed for general purpose computations.
• Obviously, the fact that computing and communication were carried
out with moving mechanical parts greatly limited the computing speed
and reliability of mechanical computers.
• Modern computers were marked by the introduction of electronic
components.
• The moving parts in mechanical computers were replaced by high-
mobility electrons in electronic computers.
• Information transmission by mechanical gears or levers was replaced
by electric signals traveling almost at the speed of light.
 Computer Generations
• Over the past several decades, electronic computers have gone through roughly five generations of
development.
• The table below provides summary of the five generations of electronic computer development.
• Each of the first three generations lasted about 10 years. The fourth generation covered a time span of 15 years.
The fifth generation today has processors and memory devices with more than 1 billion transistors on a single
silicon chip.

Generation Technology and Architecture Software and Applications Representative Systems

First (1945-1954) Vacuum tubes and relay Machine assembly languages, ENIAC, Princeton IAS, IBM
memories, CPU driven by PC single user, no subroutine 701.
and accumulator, fixed-point linkage, programmed l/O using
arithmetic. CPU.

Second (1955-1964) Discrete transistors and core IILL used with compilers, IBM T090,
memories, floating-point subroutine libraries, batch CDC I-I5-D4,
arithmetic, I/O processors, processing monitor. Univac LARC.
multiplexed memory access.
Generation Technology and Architecture Software and Applications Representative Systems

Third (1965-1974) Integrated circuits (SS1/- Multiprogramming and IBM 360/370, CDC 6600, TI-
MSI), microprograrnrning, timesharing OS, multiuser ASC, PDP-8.
pipelining, cache, and applications.
lookahead processors.

Fourth (1975-1990) LSI/ VLSI and semiconductor Multiprocessor OS, languages, VAX 9000, Cray X-MP, IBM
memory, multiprocessors, compilers, and environments 3090, BBN TC2000.
vector supercomputers, for parallel processing.
multicomputer.

Fifth (1991- present) Advanced VLSl processors, Superscalar processors, systems Desktop, Laptop, Notebooks,
memory, and switches, high- on a chip, massively parallel IBM’s Watson.
density packaging, scalable processing, grand challenge
architectures. applications, heterogeneous
processing.
 2. Elements of Modern Computers
• Hardware, software, and
programming elements of a
modern computer system are
briefly introduced below in the
context of parallel processing.
• The concept of computer
architecture is no longer
restricted to the structure of the
bare machine hardware.
• A modern computer is an
integrated system consisting of
machine hardware, an
instruction set, system software,
application programs, and user
interfaces.
Computing Problems
• The use of a computer is driven by real-life problems demanding cost effective solutions.
• Depending on the nature of the problems, the solutions may require different computing
resources.
• For numerical problems in science and technology, the solutions demand complex
mathematical formulations and intensive integer or floating-point computations.
• For alphanumerical problems in business and government, the solutions demand efficient
transaction processing, large database management, and information retrieval operations.
• For artificial intelligence (Al) problems, the solutions demand logic inferences and
symbolic manipulations.
• These computing problems have been labeled numerical computing, transaction
processing and logical reasoning.
• Some complex problems may demand a combination of these processing modes.
Algorithms and Data Structures
• Special algorithms and data structures are needed to specify the computations and
communications involved in computing problems.
• Most numerical algorithms are deterministic, using regularly structured data. Symbolic
processing may use heuristics or nondeterministic searches over large knowledge bases.
Hardware Resources
• Processors, memory, and peripheral devices form the hardware core of a computer system.
• Special hardware interfaces are often built into l/O devices such as display terminals,
workstations, optical page scanners, magnetic ink character recognizers, modems. network
adaptors, voice data entry, printers, and plotters.

In addition, software interface programs are needed. These

software interfaces include file transfer systems, editors, word
processors, device drivers, interrupt handlers, network
communication programs, etc. These programs greatly facilitate the
portability of user programs on different machine architectures.
Operating System

• An effective operating system manages the allocation and deallocation of

resources during the execution of user programs.
Beyond the OS, application software must be developed to benefit the users.
Standard benchmark programs are needed for performance evaluation.
Mapping is a bidirectional process matching algorithmic structure with hardware
architecture, and vice versa. Efficient mapping will benefit the programmer and produce
better source codes. The mapping of algorithmic and data structures onto the machine
architecture includes processor scheduling, memory maps, inter-processor communications,
etc. These activities are usually architecture-dependent.
Optimal mappings are sought for various computer architectures. The implementation of
these mappings relies on efficient compiler and operating system support.
Parallelism can be exploited at algorithm design time, at program time, at compile time,
and at run time. Techniques for exploiting parallelism at these levels form the core of
parallel processing technology.
System Software Support
• Software support is needed for the development of efficient programs in high-level
languages.
The source code written in a HLL must be first translated into object code by an
optimizing compiler.
The compiler assigns variables to registers or to memory words, and generates machine
operations corresponding to HLL operators, to produce machine code which can be
recognized by the machine hardware.
A loader is used to initiate the program execution through the OS kernel.
Resource binding demands the use of the compiler, assembler, loader and OS kernel to
commit physical machine resources to program execution. The effectiveness of this
process determines the efficiency of hardware utilization and the programmability of the
computer.
Compiler Support
• There are three compiler upgrade approaches: preprocessor, precompiler, and
parallelizing compiler.
• A preprocessor uses a sequential compiler and a low-level library of the
target computer to implement high-level parallel constructs.
• The precompiler approach requires some program flow analysis, dependence
checking, and limited optimizations toward parallelism detection.
• The third approach demands a fully developed parallelizing or vectorizing
compiler which can automatically detect parallelism in source code and
transform sequential codes into parallel constructs.
• The efficiency of the binding process depends on the effectiveness of the
preprocessor, the precompiler, the parallelizing compiler, the loader, and the
OS support.
Evolution of Computer Architecture

• The study of computer architecture involves both hardware organization and programming/
software requirements.
• As seen by an assembly language programmer, computer architecture is abstracted by its
instruction set, which includes opcode (operation codes), addressing modes, registers,
virtual memory, etc.
• From the hardware implementation point of view, the abstract machine is organized with
CPUs, caches, buses, microcode, pipelines, physical memory, etc.
• Therefore, the study of architecture covers both instruction set architectures and machine
implementation organizations.
• Over the past decades,
computer architecture has gone
through evolutional rather than
revolutional changes.
• As depicted in the figure, we
started with the von Neumann
architecture built as a
sequential machine executing
scalar data.
• The sequential computer was
improved from bit-serial to
word—parallel operations, and
from fixed—point to floating
point operations.
• The von Neumann architecture
is slow due to sequential
execution of instructions in
programs.
• Lookahead techniques were introduced to prefetch instructions in order to overlap I/E
(Instruction Fetch/ Decode) operations and to enable functional parallelism.
• Functional parallelism was supported by two approaches: One is to use multiple functional
units simultaneously, and the other is to practice pipelining at various processing levels.
• The latter includes pipelined instruction execution, pipelined arithmetic computations, and
memory-access operations.
• Pipelining is a technique where multiple instructions are overlapped during execution.
• Pipelining has proven especially attractive in performing identical operations repeatedly
over vector data strings.
• Vector operations were originally carried out implicitly by software-controlled looping
using scalar pipeline processors.
ARCHITECTUR
AL
CLASSIFICATIO
N
 Basic types of architectural classification
 FLYNN’S TAXONOMY OF COMPUTER ARCHITECTURE
 FENG’S CLASSIFICATION
 Handler Classification
 Other types of architectural classification
 Classification based on coupling between processing
elements
 Classification based on mode of accessing memory
 ARCHITECTURAL CLASSIFICATION
 Flynn’s classification: (1966) is based on
multiplicity of instruction streams and the data streams
in computer systems.

 Feng’s classification: (1972) is based on serial

versus parallel processing.

 Handler’s classification: (1977) is determined by

the degree of parallelism and pipelining in various
subsystem levels.
FLYNN’S TAXONOMY OF
ARCHITECTURE… COMPUTER
 The most popular taxonomy of computer
architecture was defined by Flynn in 1966.

 Flynn’s classification scheme is based on the notion of a

stream of information. Two types of information flow into a
processor: instructions and data.

 The instruction stream is defined as the sequence of

instructions performed by the processing unit.

 The data stream is defined as the data traffic

exchanged between the memory and
processing unit. the
 Types of FLYNN’S TAXONOMY
 to Flynn’s classification, either of the
instruction or data streams can be
single or multiple. Computer
architecture can be classified into
the following four distinct categories:

 single-instruction single-data
streams (SISD);
 single-instruction multiple-data
streams (SIMD);
 multiple-instruction single-data
streams (MISD); and
 multiple-instruction multiple-
data streams (MIMD).
 SISD
 Conventional single-processor von Neumann
computers are classified as SISD systems.
 SIMD ARCHITECTURE
 The SIMD model of parallel
computing consists of two
parts: a front-end computer of
the usual von Neumann style,
and a processor array.

 The processor array is a set of

identical synchronized
processing elements
capable of simultaneously
performing the same
operation on different data.
 SIMD ARCHITECTURE
 Each processor in the array has
a small amount of local
memory where the
distributed data resides
while it is being processed
in parallel.

 The processor array is

connected to the memory bus
of the front end so that the front
end can randomly access the
local processor memories as
if it were another memory.
 SIMD ARCHITECTURE
 The front end can issue
special commands that
cause parts of the memory to
be operated on
simultaneously or cause
data to move around in the
memory.

 The application program is

executed by the front end in
the usual serial way, but issues
commands to the processor
array to carry out SIMD
operations in parallel.
 There are two main configurations that
have been used in SIMD machines.
In the first scheme, each processor has its own local
memory. Processors can communicate with each other
through the interconnection network.
 There are two main configurations that
have been used in SIMD machines.
second SIMD scheme, processors and memory
modules communicate with each other via the
interconnection network.
 MIMD ARCHITECTURE
 Multiple-instruction multiple-data streams
(MIMD) parallel architectures are made of
multiple processors and multiple memory
modules connected together via some
interconnection network. They fall into two broad
categories: shared memory or message
passing.
 Processors exchange information through their
central shared memory in shared memory
systems, and exchange information through their
interconnection network in message passing
systems.
MIMD
 MIMD “shared memory system“
 A shared memory system typically
accomplishes inter-processor coordination
through a global memory shared by all
processors.

 Because access to shared memory is

balanced, these systems are also called SMP
(symmetric multiprocessor) systems.
 MIMD “message passing system”
 A message passing system (also referred to as
distributed memory) typically combines the local
memory and processor at each node of the
interconnection network.
 There is no global memory, so it is necessary to move
data from one local memory to another by
means of message passing.
 This is typically done by a Send/Receive pair of
commands, which must be written into the application
software by a programmer.
 MISD ARCHITECTURE
 In the MISD category, the
same stream of data
flows through a linear
array of processors
executing different
instruction streams.
 In practice, there is no
viable MISD machine;
however, some authors have
considered pipelined
machines (and
perhaps systolic-array
computers) as examples
for MISD.
 FENG’S CLASSIFICATION
 Tse-yun Feng suggested the use of degree of parallelism to classify
various computer architectures.

 The maximum number of binary digits that can be

processed within a unit time by a computer system is called the maximum parallelism degree

 A bit slice is a string of bits one from each of the words at the same vertical position.

 under above classification

 Word Serial and Bit Serial (WSBS)
 Word Parallel and Bit Serial (WPBS)
 Word Serial and Bit Parallel(WSBP)
 Word Parallel and Bit Parallel (WPBP)
 WSBS has been called bit parallel processing
because one bit is processed at a time.

 WPBS has been called bit slice processing because m-bit

slice is processes at a time.

 WSBP is found in most existing computers and has been

called as Word Slice processing because one word of n
bit processed at a time.

 WPBP is known as fully parallel processing in which

an array on n x m bits is processes at one time.
 Handler Classification
 Wolfgang Handler has proposed a classification scheme
for identifying the parallelism degree and pipelining degree
built into the hardware structure of a computer system. He
considers at three subsystem levels:
 Processor Control Unit (PCU)
 Arithmetic Logic Unit (ALU)
 Bit Level Circuit (BLC)

 Each PCU corresponds to one processor or one CPU. The

ALU is equivalent to Processor Element (PE). The BLC
corresponds to combinational logic circuitry needed to
perform 1 bit operations in the ALU.
 Classification based on coupling
between processing elements
Coupling refers to the way in which PEs cooperate with one
another.
 Loosely coupled: the degree of coupling between the PEs is less.
Example: parallel computer consisting of workstations
connected together by local area network such as Ethernet is loosely
coupled. In this case each one of the workstations works
independently.
If they want to cooperate they will exchange message. Thus logically
they are autonomous and physically they do not share any memory
and communication via I/O channels.

 Tightly coupled: a tightly coupled parallel computer, on the other

hand shares a common main memory. Thus communication
among PEs is very fast and cooperation may be even at the
level of instructions carried out by each PE as they share a
common memory.
 Classification based on mode
of accessing memory
 Uniform memory access parallel computers (UMC): in a shared
memory computer system all processors share a common global
address space. For these systems the time to access a work in
memory is constant for all processors. Such a parallel computer is
said to have a Uniform Memory Access (UMA).

 Non uniform memory access parallel computers: in a

distributed shared memory computer system, each processor
may its own local memory and may or may not share a
common memory. For these systems, the time taken to access
a word in local memory smaller than the time taken to access a
word stored in memory of other computer or common shared
memory. thus this systems said to have Non Uniform Memory
Access (NUMA)
Pipeline and its Principles
• Pipelining is a technique of decomposing a sequential process into suboperations, with
each subprocess being executed in a special dedicated segment that operates concurrently
with all other segments.
• The overlapping of computation is made possible by associating a register with each
segment in the pipeline.
• The registers provide isolation between each segment so that each can operate on distinct
data simultaneously.
• Perhaps the simplest way of viewing the pipeline structure is to imagine that each segment
consists of an input register followed by a combinational circuit.
• The register holds the data.
• The combinational circuit performs the suboperation in the particular segment.

• A clock is applied to all registers after enough time has elapsed to perform all segment
activity.
• The pipeline organization is demonstrated by means of a simple example.
• To perform the combined multiply and add operations with a stream of numbers
Ai * Bi + Ci for i = 1, 2, 3, …, 7
• Each suboperation is to be implemented in a segment within a pipeline.
R1 ← Ai , R2 ← Bi Input Ai and Bi
R3 ← R1 * R2, R4 ← Ci Multiply and input Ci
R5 ← R3 + R4 Add Ci to product
• Each segment has one or two registers and a combinational circuit as shown in Fig below.
• The five registers are loaded with new data every clock pulse. The effect of each clock is
shown in the table.
 General Considerations
• Any operation that can be decomposed into a sequence of suboperations of about the same
complexity can be implemented by a pipeline processor.
• The general structure of a four-segment pipeline is illustrated in Fig below.

• We define a task as the total operation performed going through all the segments in the
pipeline.
• The behavior of a pipeline can be illustrated with a space-time diagram.
• It shows the segment utilization as a function of time.

• The space-time diagram of a four-segment pipeline is demonstrated in Fig below.

• Let us consider a case where a k-segment pipeline with a clock cycle time tp is used to
execute n tasks.
• The first task T1 requires a time equal to ktp to complete its operation.
• The remaining n-1 tasks will be completed after a time equal to (n-1)tp.
• Therefore, to complete n tasks using a k-segment pipeline requires k+(n-1) clock cycles.

• Now, consider a non-pipeline unit that performs the same operation and takes a time equal
to tn to complete each task.
• The total time required for n tasks is ntn.

• The speedup of a pipeline processing over an equivalent non-pipeline processing is defined

by the ratio:
S = ntn/(k+n-1)tp
• As the number of tasks increases, n becomes much larger than k-1, and k+n-1 approaches
the value of n. Under this condition, the speedup becomes
S = tn/tp.
• If we assume that the time it takes to process a task is the same in the pipeline and non-
pipeline circuits, i.e., tn = ktp, the speedup reduces to
S = ktp / tp= k.
• This shows that the theoretical maximum speed up that a pipeline can provide is k, where
k is the number of segments in the pipeline.
Implicit Parallelism
• An intrinsic approach uses a conventional
language, such as C, Fortran, Lisp or Pascal,
to write the source program. The
sequentially coded source program is
translated into parallel object code by a
parallelizing compiler. As illustrated in
figure, the compiler must be able to detect
parallelism and assign target machine
resources. This compiler approach has been
applied in programming shared-memory
multiprocessors.
• With parallelism being implicit, success
relies heavily on the “intelligence” of a
parallelizing compiler. This approach
requires less effort on the part of the
programmer.
Extrinsic Parallelism
• The second approach requires more effort
by the programmer to develop a source
program using parallel dialects of C,
Fortran, Lisp, or Pascal. Parallelism is
explicitly specified in the user programs.
This will significantly reduce the burden on
the compiler to detect parallelism. Instead,
the compiler needs to preserve parallelism
and, where possible, assigns target machine
resources.