Cloud Computing & IoT

Lecture - 2

Prof. Sankhadeep Chatterjee

Department of Computer Science & Engineering, UEMK
 Reference

 Mastering Cloud Computing, 1st Edition, Foundations

and Applications Programming, Rajkumar Buyya,
Christian Vecchiola, S.Thamarai Selvi (Chapter 1)
Recap

 Over view of Cloud computing

 Motivation

 Sequential & Parallel model

 Brief
introduction to Parallel computing
model
 In this Lecture

 Principles of Parallel and Distributed

Computing (continue)
 Introduction to Parallel computing

 Parallel Processing influencing factors

 Hardware architectures for parallel

Processing
 Principles of Parallel and
Distributed Computing Contd…
 The shared memory has a single address
space, which is accessible to all the processors.

 Parallel programs are then broken down into

several units of execution that
 can be allocated to different processors and
 can communicate with each other by means of
shared memory.
 Principles of Parallel and
Distributed Computing Contd…

 The term distributed computing encompasses

 any architecture or system that
 allows the computation to be broken down into
units and
 executed concurrently on different computing
elements,
 whether these are processors on different nodes,
processors on the same computer, or cores within
the same processor.
 Principles of Parallel and
Distributed Computing Contd…
 Distributed computing includes a wider range of
systems and applications than parallel computing
and is often considered a more general term.

 Even though it is not a rule, the term distributed

often implies that
 The locations of the computing elements are not
the same
 Such elements might be heterogeneous in terms
of hardware and software features.
 Motivation behind Parallel computing

 Silicon-based processor chips are reaching

their physical limits.
 Processing speed is constrained by the speed
of light
 The density of transistors packaged in a
processor is constrained by thermodynamics
limitations.
 Motivation behind Parallel computing

 A viable solution to overcome this limitation is

to connect multiple processors working in
coordination with each other.

 As discussed earlier, the term parallel

computing has blurred its edges with the term
distributed computing and is often used in
place of later term.
 What is Parallel Processing?

 Processing of multiple tasks

simultaneously on multiple processors is
called parallel processing.

 The parallel program consists of multiple

active processes ( tasks) simultaneously
solving a given problem.
 Parallel Processing contd.

 A given task is divided into multiple subtasks

using a divide-and-conquer technique, and each
subtask is processed on a different central
processing unit (CPU).

 Programming on multi processor system using

the divide-and-conquer technique is called
parallel programming.
 Parallel Processing influencing
factors
 Computational requirements are ever increasing in the
areas of both scientific and business computing. The
technical computing problems, which require high-speed
computational power, are related to
 life sciences,
 aerospace,
 geographical information systems,
 mechanical design and analysis etc.
 Parallel Processing influencing
factors
 Sequential architectures are reaching mechanical
physical limitations as they are constrained by the speed
of light and thermodynamics laws.

 The speed which sequential CPUs can operated is

reaching saturation point ( no more vertical growth),

 Hence an alternative way to get high computation

speed is to connect multiple CPUs ( opportunity for
horizontal growth).
 Evolution of multi-core processors

Reference: Tendulkar, P. (2014). Mapping and scheduling on multi-core processors

using SMT solvers (Doctoral dissertation, Universite de Grenoble I-Joseph Fourier).
 Parallel Processing influencing
factors
 Hardware improvements in pipelining , super
scalar, and the like are non scalable and require
sophisticated compiler technology.

 However, developing such compiler

technology is a difficult task.
 Hardware architectures for
parallel Processing
 The core elements of parallel processing are CPUs.
Based on the number of instructions and data streams,
that can be processed simultaneously, computing
systems are classified into the following four
categories:
 Single-instruction, Single-data (SISD) systems
 Single-instruction, Multiple-data (SIMD) systems
 Multiple-instruction, Single-data (MISD) systems
 Multiple-instruction, Multiple-data (MIMD) systems
 Single – Instruction , Single
Data (SISD) systems
 SISD computing system is a uni-
processor machine capable of
executing a single instruction, which
operates on a single data stream. Instructio
n Stream

 Machine instructions are processed

sequentially, hence computers Data Data
adopting this model are popularly Input Output
called sequential computers.

 Most conventional computers are built Processor

using SISD model.
 Single – Instruction , Single
Data (SISD) systems
 All the instructions and data
to be processed have to be Instructi
on
stored in primary memory. Stream

Data Data
 The speed of processing Input Output
element in the SISD model is
limited by the rate at which
the computer can transfer Processor
information internally.
 Single – Instruction , Multiple
Data (SIMD) systems
Single Instruction Stream
 SIMD computing system is
a multiprocessor machine
capable of executing the
same instruction on all the Data
Input 1
Data
Output 1

CPUs but operating on

different data streams. Processor 1

 Machines based on this Data

Data
Output 2
Input 2
model are well suited for
scientific computing since Processor 2

Data
they involve lots of vector Data
Input N
Output N

and matrix operations.

Processor N
 Single – Instruction , Multiple
Data (SIMD) systems
Single Instruction Stream
 For instance statement :
 Ci = Ai * Bi, can be passed
to all the processing
Data Data
elements (PEs), Input 1 Output 1

 Organized data elements Processor 1

of vectors A and B can be

Data
divided into multiple sets ( Data
Input 2
Output 2

N- sets for N PE systems),

Processor 2

 And each PE can process Data

Data
Output N
one data set. Input N

Processor N
 Multiple – Instruction , Single
Data (MISD) systems
 MISD computing Instruction
Stream 1
Instruction
Stream 2
Instruction
Stream N

system is a multi
processor machine
capable of executing
different instructions Processor 1

Single Data Output Stream

on different PEs all of Single Data Input Stream

them operating on
the same data set. Processor 2

 For example
Processor N
 y = sin(x) + cos(x) +
tan(x)
 Multiple – Instruction , Single
Data (MISD) systems
 Machines built using
Instruction Instruction Instruction
MISD model are not Stream 1 Stream 2 Stream N

useful in most of the

applications.
 Few machines are
built but none of Processor 1

Single Data Output Stream

Single Data Input Stream
them available
commercially.
Processor 2

 This type of systems

are more of an
intellectual exercise Processor N

than a practical
configuration.
 Multiple – Instruction ,
Multiple Data (MIMD) systems
Instruction Instruction Instruction
 MIMD computing system Stream 1 Stream 2 Stream N

is a multi processor
machine capable of
executing multiple Data Input 1 Data Output 1
instructions on multiple
data sets.
Processor 1

 Each PE in the MIMD Data Input 2 Data Output 2

model has separate
instruction and data
Processor 2
streams, hence machines
built using this model are Data Input N Data Output 3

well suited to any kind of

application.
Processor N
 Multiple – Instruction ,
Multiple Data (MIMD) systems
Instruction Instruction Instruction
 Unlike SIMD, MISD Stream 1 Stream 2 Stream N

machine, PEs in MIMD

machines work
asynchronously, Data Input 1 Data Output 1

 MIMD machines are Processor 1

broadly categorized into
 shared-memory MIMD Data Input 2 Data Output 2
 distributed memory MIMD

Processor 2
 Based on the way PEs
are coupled to the main Data Input N Data Output 3

memory.
Processor N
 Shared Memory MIMD machines

 All the PEs are connected to a single global

memory and they all have access to it.
 Systems based on this model are also
called tightly coupled multi processor
systems.
Processor 1 Processor 2 Processor N
 The communication between PEs in this
model takes place through the shared Memory
memory. Bus

 Modification of the data stored in the global

memory by one PE is visible to all other Global System Memory
PEs.
 Dominant representative shared memory
MIMD systems are silicon graphics
machines and Sun/IBM SMP ( Symmetric
Multi-Processing).
 Distributed Memory MIMD machines
IPC Channel IPC Channel
 All PEs have a local memory. Systems
based on this model are also called
loosely coupled multi processor
systems.
 The communication between PEs in
this model takes place through the
interconnection network, the inter Processor 1 Processor 2 Processor 2
process communication channel, or
IPC. Memory Memory Memory
Bus Bus Bus
 The network connecting PEs can be
configured to tree, mesh, cube, and so
on. Local Local Local
Memory Memory Memory
 Each PE operates asynchronously,
and if communication/synchronization
among tasks is necessary, they can do
so by exchanging messages between
them.
 Shared Vs Distributed MIMD model

 The shared memory MIMD architecture is easier to program but is less

tolerant to failures and harder to extend with respect to the distributed
memory MIMD model.

 Failures, in a shared memory MIMD affect the entire system, whereas this is
not the case of the distributed model, in which each of the PEs can be easily
isolated.

 Moreover, shared memory MIMD architectures are less likely to scale

because the addition of more PEs leads to memory contention.

 This is a situation that does not happen in the case of distributed memory,
in which each PE has its own memory.

 As a result, distributed memory MIMD architectures are most popular today.

Thank You