Distributed Systems

Unit I

1.1 Definition:
A distributed system is one in which components located at networked computers communicate
and coordinate their actions only by passing messages. This definition leads to the following
especially significant characteristics of distributed systems: concurrency of components, lack of
a global clock and independent failures of components.

Features
 No common physical clock
 No shared memory
 Geographical separation- Recently, the network/cluster of workstations (NOW/COW)
configuration connecting processors on a LAN is also being increasingly regarded as a
small distributed system. The Google search engine is based on the NOW architecture.
 Autonomy and heterogeneity

1.2 Relation to computer system components

A typical distributed system is shown in Figure 1.1. Each computer has a memory-processing
unit and the computers are connected by a communication network. Figure 1.2 shows the
relationships of the software components that run on each of the computers and use the local
operating system and network protocol stack for functioning. The distributed software is also
termed as middleware. A distributed execution is the execution of processes across the
distributed system to collaboratively achieve a common goal. An execution is also sometimes
termed a computation or a run.

The distributed system uses a layered architecture to break down the complexity of system
design. The middleware is the distributed software that drives the distributed system, while
providing transparency of heterogeneity at the platform level. Figure 1.2 schematically shows the
interaction of this software with these system components at each processor. There are several
middleware standards such as Object Management Group’s (OMG) common object request
broker architecture (CORBA), and the remote procedure call (RPC) mechanism.
1.3 Motivation
 Inherently distributed computations
 Resource sharing
 Access to geographically remote data and resources
 Enhanced reliability
o availability
o integrity
o fault-tolerance
 Increased performance/cost ratio
 Scalability
 Modularity and incremental expandability
1.4 Relation to parallel multiprocessor/multicomputer systems
Multiprocessor Operating System refers to the use of two or more central processing
units (CPU) within a single computer system. These multiple CPUs are in a close
communication sharing the computer bus, memory and other peripheral devices. These systems
are referred as tightly coupled systems.
Parallel Processing Systems are designed to speed up the execution of programs by
dividing the program into multiple fragments and processing these fragments simultaneously.
Such systems are multiprocessor systems also known as tightly coupled systems.
1.4.1 Characteristics of parallel systems
A parallel system may be broadly classified as belonging to one of three types:

 1. A multiprocessor system is a parallel system in which the multiple processors have

direct access to shared memory which forms a common address space. The architecture
is shown in Figure 1.3(a). Such processors usually do not have a common clock. A
multiprocessor system usually corresponds to a uniform memory access (UMA)
architecture in which the access latency, i.e., waiting time, to complete an access to any
memory location from any processor is the same. The processors are in very close
physical proximity and are connected by an interconnection network. Interprocess
communication across processors is traditionally through read and writes operations on
the shared memory, although the use of message-passing primitives such as those
provided by the MPI, is also possible (using emulation on the shared memory). All the
 processors usually run the same operating system, and both the hardware and software
are very tightly coupled.

Figure 1.4 shows two popular interconnection networks – the Omega network [4] and the
Butterfly network [10], each of which is a multi-stage network formed of 2×2 switching
elements. Each 2×2 switch allows data on either of the two input wires to be switched to the
upper or the lower output wire. In a single step, however, only one data unit can be sent on an
output wire. So if the data from both the input wires is to be routed to the same output wire in a
single step, there is a collision. Various techniques such as buffering or more elaborate
interconnection designs can address collisions.

An 8x8 Omega network is a multistage interconnection network, meaning that processing

elements (PEs) are connected using multiple stages of switches. Inputs and outputs are given
with addresses. The outputs from each stage are connected to the inputs of the next stage using a
perfect shuffle connection system. This means that the connections at each stage represent the
movement of a deck of cards divided into 2 equal decks and then shuffled together, with each
card from one deck alternating with the corresponding card from the other deck. In terms of
binary representation of the PEs, each stage of the perfect shuffle can be thought of as a cyclic
logical left shift; each bit in the address is shifted once to the left, with the most significant bit
moving to the least significant bit.

Destination-tag routing
In destination-tag routing, switch settings are determined solely by the message destination. The
most significant bit of the destination address is used to select the output of the switch in the first
stage; if the most significant bit is 0, the upper output is selected, and if it is 1, the lower output is
selected. The next-most significant bit of the destination address is used to select the output of
the switch in the next stage, and so on until the final output has been selected.

For example, if a message's destination is PE 001, the switch settings are: upper, upper, lower. If
a message's destination is PE 101, the switch settings are: lower, upper, lower. These switch
settings hold regardless of the PE sending the message

XOR-tag routing

In XOR-tag routing, switch settings are based on (source PE) XOR (destination PE). This XOR-
tag contains 1s in the bit positions that must be swapped and 0s in the bit positions that both
source and destination have in common. The most significant bit of the XOR-tag is used to select
the setting of the switch in the first stage; if the most significant bit is 0, the switch is set to pass-
through, and if it is 1, the switch is crossed. The next-most significant bit of the tag is used to set
the switch in the next stage, and so on until the final output has been selected.

For example, if PE 001 wishes to send a message to PE 010, the XOR-tag will be 011 and the
appropriate switch settings are: A2 straight, B3 crossed, C2 crossed.

A butterfly network is a technique to link multiple computers into a high-speed network. This
form of multistage interconnection network topology can be used to connect different nodes in a
multiprocessor system.

In fig 1.5 b Thus, the processors are labelled such that the shortest path between any two
processors is the Hamming distance (defined as the number of bit positions in which the
two equal sized bit strings differ) between the processor labels.
 2. A multicomputer parallel system is a parallel system in which the multiple processors
do not have direct access to shared memory. The memory of the multiple processors may
or may not form a common address space. Such computers usually do not have a
common clock. The architecture is shown in Figure 1.3(b).
  3. Array processors belong to a class of parallel computers that are physically co-located,
are very tightly coupled, and have a common system clock (but may not share memory
and communicate by passing data using messages). Array processors and systolic arrays
that perform tightly synchronized processing and data exchange in lock-step for
applications such as DSP and image processing belong to this category. These
applications usually involve a large number of iterations on the data. This class of parallel
systems has a very niche market.

Let’s see the difference between UMA and NUMA:

S.NO UMA NUMA

UMA stands for Uniform Memory NUMA stands for Non-uniform
1.
Access. Memory Access.
In Uniform Memory Access, Single In Non-uniform Memory Access,
2.
memory controller is used. Different memory controller is used.
Uniform Memory Access is slower Non-uniform Memory Access is faster
3.
than non-uniform Memory Access. than uniform Memory Access.
Non-uniform Memory Access has more
Uniform Memory Access has limited
4. bandwidth than uniform Memory
bandwidth.
Access.
Uniform Memory Access is applicable Non-uniform Memory Access is
5. for general purpose applications and applicable for real-time applications and
time-sharing applications. time-critical applications.
In uniform Memory Access, memory In non-uniform Memory Access,
6.
access time is balanced or equal. memory access time is not equal.
There are 3 types of buses used in While in non-uniform Memory Access,
7. uniform Memory Access which are: There are 2 types of buses used which
Single, Multiple and Crossbar. are: Tree and hierarchical.

1.4.2 Flynn’s taxonomy

Flynn identified four processing modes, based on whether the processors execute the
same or different instruction streams at the same time, and whether or not the processors
processed the same (identical) data at the same time. It is instructive to examine this
classification to understand the range of options used for configuring systems:
• Single instruction stream, single data stream (SISD)
This mode corresponds to the conventional processing in the von Neumann paradigm with a
single CPU, and a single memory unit connected by a system bus.
• Single instruction stream, multiple data stream (SIMD)
This mode corresponds to the processing by multiple homogenous processors which
execute in lock-step on different data items. Applications that involve operations on large arrays
and matrices, such as scientific applications, can best exploit systems that provide the SIMD
mode of operation because the data sets can be partitioned easily. Several of the earliest parallel
computers, such as Illiac-IV, MPP, CM2, and MasPar MP-1 were SIMD machines. Vector
processors, array processors’ and systolic arrays also belong to the SIMD class of processing.
Recent SIMD architectures include co-processing units such as the MMX units in Intel
processors (e.g., Pentium with the streaming SIMD extensions
(SSE) options) and DSP chips such as the Sharc [22].
• Multiple instruction stream, single data stream (MISD)
This mode corresponds to the execution of different operations in parallel on the same
data. This is a specialized mode of operation with limited but niche applications, e.g.,
visualization.
• Multiple instruction stream, multiple data stream (MIMD)
In this mode, the various processors execute different code on different data. This is the
mode of operation in distributed systems as well as in the vast majority of parallel systems.
There is no common clock among the system processors. Sun Ultra servers, multicomputer PCs,
and IBM SP machines are examples of machines that execute in MIMD mode.

1.4.3 Coupling, parallelism, concurrency, and granularity

The degree of coupling among a set of modules, whether hardware or software, is
measured in terms of the interdependency and binding and/or homogeneity among the modules.
When the degree of coupling is high (low), the modules are said to be tightly (loosely) coupled.
SIMD and MISD architectures generally tend to be tightly coupled because of the common
clocking of the shared instruction stream or the shared data stream.

Concurrent processing is a computing model in which multiple processors execute

instructions simultaneously for better performance. Concurrent means something that happens
at the same time as something else. ... Concurrent processing is sometimes said to be
synonymous with parallel processing.

In parallel computing, granularity (or grain size) of a task is a measure of the amount of
work (or computation) which is performed by that task.

Another definition of granularity takes into account the communication overhead between
multiple processors or processing elements. It defines granularity as the ratio of computation
time to communication time, wherein, computation time is the time required to perform the
computation of a task and communication time is the time required to exchange data between
processors.

If Tcomp is the computation time and Tcomm denotes the communication time, then the
Granularity G of a task can be calculated as:

G= Tcomp/ Tcomm
Granularity is usually measured in terms of the number of instructions executed in a particular
task. Alternately, granularity can also be specified in terms of the execution time of a program,
combining the computation time and communication time.

1.5 Message-passing systems versus shared memory systems

Shared memory systems are those in which there is a (common) shared address space
throughout the system. Communication among processors takes place via shared data variables,
and control variables for synchronization among the processors. Semaphores and monitors that
were originally designed for shared memory uniprocessors and multiprocessors are examples of
how synchronization can be achieved in shared memory systems. For a distributed system, this
abstraction is called distributed shared memory.
1.5.1 Emulating message-passing on a shared memory system (MP →SM)
The shared address space can be partitioned into disjoint parts, one part being assigned to
each processor. “Send” and “receive” operations can be implemented by writing to and reading
from the destination/sender processor’s address space, respectively,
1.5.2 Emulating shared memory on a message-passing system (SM →MP)
This involves the use of “send” and “receive” operations for “write” and “read”
operations. Each shared location can be modeled as a separate process; “write” to a shared
location is emulated by sending an update message to the corresponding owner process; a “read”
to a shared location is emulated by sending a query message to the owner process. As accessing
another processor’s memory requires send and receives operations, this emulation is expensive.
1.6 Primitives for distributed communication
1.6.1 Blocking/non-blocking, synchronous/asynchronous primitives
Message send and message receive communication primitives are denoted Send() and
Receive(), respectively. A Send primitive has at least two parameters – the destination, and the
buffer in the user space, containing the data to be sent. Similarly, a Receive primitive has at least
two parameters – the source from which the data is to be received (this could be a wildcard), and
the user buffer into which the data is to be received.
There are two ways of sending data when the Send primitive is invoked – the buffered option and
the unbuffered option. The buffered option which is the standard option copies the data from the
user buffer to the kernel buffer. The data later gets copied from the kernel buffer onto the
network. In the unbuffered option, the data gets copied directly from the user buffer onto the
network. For the Receive primitive, the buffered option is usually required because the data may
already have arrived when the primitive is invoked, and needs storage place in the kernel.
The following are some definitions of blocking/non-blocking and synchronous/asynchronous
primitives
 Synchronous primitives
 Asynchronous primitives
 Blocking primitives
 Non-blocking primitives
Blocking synchronous Send
non-blocking synchronous Send
Blocking asynchronous Send
non-blocking asynchronous Send
Blocking Receive
non-blocking Receive

1.6.2 Processor synchrony

Processor synchrony indicates that all the processors execute in lock-step with their
clocks synchronized.
1.6.3 Libraries and standards
Many commercial software products (banking, payroll, etc., applications) use
proprietary primitive libraries supplied with the software marketed by the vendors (e.g., the IBM
CICS software which has a very widely installed customer base worldwide uses its own
primitives). The message-passing interface (MPI) library and the PVM (parallel virtual machine)
library are used largely by the scientific community, but other alternative libraries exist.
Commercial software is often written using the remote procedure calls (RPC) mechanism in
which procedures that potentially reside across the network are invoked transparently to the user,
in the same manner that a local procedure is invoked
With the growth of object based software, libraries for remote method invocation (RMI)
and remote object invocation (ROI) with their own set of primitives are being proposed and
standardized by different agencies. CORBA (common object request broker architecture) and
DCOM (distributed component object model) are two other standardized architectures with their
own set of primitives.
1.7 Synchronous versus asynchronous executions
In addition to the two classifications of processor synchrony/asynchrony and of synchronous/
asynchronous communication primitives, there is another classification, namely that of
synchronous/asynchronous executions
 An asynchronous execution is an execution in which (i) there is no processor synchrony
and there is no bound on the drift rate of processor clocks, (ii) message delays
(transmission + propagation times) are finite but unbounded, and (iii) there is no upper
bound on the time taken by a process to execute a step.

 A synchronous execution is an execution in which (i) processors are synchronized (ii)

message delivery (transmission + delivery) times are such that they occur in one logical
step or round, and (iii) there is a known upper bound on the time taken by a process to
execute a step.
1.7.1 Emulating an asynchronous system by a synchronous system (A→S)
An asynchronous program (written for an asynchronous system) can be emulated on a
synchronous system fairly trivially as the synchronous system is a special case of an
asynchronous system – all communication finishes within the same round in which it is initiated
1.7.2 Emulating a synchronous system by an asynchronous system (S →A)
A synchronous program (written for a synchronous system) can be emulated on an
asynchronous system using a tool called synchronizer.
1.7.3 Emulations

1.8 Design issues and challenges

1.8.1 Distributed systems challenges from a system perspective

Communication
Processes
Naming
Synchronization
Data storage and access
Consistency and replication
Fault tolerance

 Detecting failures
 Masking failures
 Tolerating failures:
 Recovery from failures
 Redundancy

Security

 Denial of service attacks

 Security of mobile code
Applications Programming Interface (API) and transparency

 Access transparency
 Location transparency
 Concurrency transparency
 Replication transparency
 Failure transparency
 Mobility transparency
 Performance transparency
 Scaling transparency
Scalability and modularity

1.8.2 Algorithmic challenges in distributed computing

Designing useful execution models and frameworks

Dynamic distributed graph algorithms and distributed routing algorithms

Time and global state in a distributed system

Synchronization/coordination mechanisms

 Physical clock synchronization

 Leader election
 Mutual exclusion
 Deadlock detection and resolution
 Termination detection
 Garbage collection

Group communication, multicast and ordered message delivery

Monitoring distributed events and predicates

Distributed program design and verification tools

Debugging distributed programs

Data replication, consistency models, and caching

World Wide Web design – caching, searching, scheduling

Distributed shared memory abstraction

 Wait-free algorithms
 Mutual exclusion
 Register constructions
 Consistency models

Reliable and fault-tolerant distributed systems

 Consensus algorithms
 Replication and replica management
 Voting and quorum systems
 Distributed databases and distributed commit
 Self-stabilizing systems
 Checkpointing and recovery algorithms
  Failure detectors

Load balancing

 Data migration
 Computation migration
 Distributed scheduling

Real-time scheduling

Performance

 Metrics
 Measurement methods/tools

1.8.3 Applications of distributed computing and newer challenges

Mobile systems

The portability of many of these devices, together with their ability to connect conveniently to
networks in different places, makes mobile computing possible. Mobile computing is the
performance of computing tasks while the user is on the move, or visiting places other than their
usual environment. In mobile computing, users who are away from their ‘home’ intranet (the
intranet at work, or their residence) are still provided with access to resources via the devices
they carry with them. They can continue to access the Internet; they can continue to access
resources in their home intranet; and there is increasing provision for users to utilize resources
such as printers or even sales points that are conveniently nearby as they move around. The latter
is also known as location-aware or context-aware computing. Mobility introduces a number of
challenges for distributed systems, including the need to deal with variable connectivity and
indeed disconnection, and the need to maintain operation in the face of device mobility

Sensor networks

A wireless sensor network (WSN) is a wireless network consisting of spatially distributed

autonomous devices using sensors to monitor physical or environmental conditions. A WSN
system incorporates a gateway that provides wireless connectivity back to the wired world and
distributed nodes 

The Internet of Things (IoT) refers to a network comprised of physical objects capable of
gathering and sharing electronic information. The Internet of Things includes a wide variety of
“smart” devices, from industrial machines that transmit data about the production process to
sensors that track information about the human body.

Ubiquitous or pervasive computing

Ubiquitous computing is a concept in software engineering and computer science where
computing is made to appear anytime and everywhere. In contrast to desktop computing,
ubiquitous computing can occur using any device, in any location, and in any format.

Peer-to-peer computing

A peer-to-peer network allows computer hardware and software to communicate without the
need for a server. Unlike client-server architecture, there is no central server for processing
requests in a P2P architecture. The peers directly interact with one another without the
requirement of a central server.

Publish-subscribe, content distribution, and multimedia

Distributed agents

Distributed data mining

Grid computing

Cluster computing or High-Performance computing frameworks is a form of computing

in which bunch of computers (often called nodes) that are connected through a LAN (local area
network) so that, they behave like a single machine. A computer cluster help to solve complex
operations more efficiently with much faster processing speed, better data integrity than a single
computer and they only used for mission-critical applications.

Grid computing is a group of computers physically connected (over a network or with

Internet) to perform a dedicated tasks together, such as analysing e-commerce data and solve a
complex problem. Grids are a form of "super virtual computer" that solve a particular
application. The grid size may vary from small to large enterprises network.

Cloud computing is a general term for anything that involves delivering hosted services
over the Internet. These services are broadly divided into three categories: Infrastructure-as-a-
Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS). The name cloud
computing was inspired by the cloud symbol that's often used to represent the Internet in
flowcharts and diagrams.

Security in distributed systems