lOMoARcPSD|19961013

CS3551 Unit II Notes - none

Distributed computing (Anna University)

Studocu is not sponsored or endorsed by any college or university

Downloaded by akileshwari Gopalakrishnan ([email protected])
 lOMoARcPSD|19961013

UNIT II
LOGICAL TIME AND GLOBAL STATE

Logical Time: Physical Clock Synchronization: NTP – A Framework for a System of Logical Clocks
– Scalar Time – Vector Time; Message ordering and group communication: Message ordering
paradigms –Asynchronous execution with synchronous communication –Synchronous
program order on an asynchronous system –Group communication – Causal order (CO) –
Total order. Global state and snapshot recording algorithms: Introduction –System model and
definitions –Snapshot algorithms for FIFO channels.

LOGICAL TIME
Logical clocks are based on capturing chronological and causal relationships of processes and
ordering events based on these relationships.

Precise physical clocking is not possible in distributed systems. The

asynchronous distributed systems spans logical clock for coordinating the events.Three
types of logical clock are maintained in distributed systems:
 Scalar clock
 Vector clock
 Matrix clock

In a system of logical clocks, every process has a logical clock that is advanced using
a set of rules. Every event is assigned a timestamp and the causality relation between
events can be generally inferred from their timestamps.
The timestamps assigned to events obey the fundamental monotonicity property; that
is, if an event a causally affects an event b, then the timestamp of a is smaller than the
timestamp of b.

Differences between physical and logical clock

Physical Clock Logical Clock
A physical clock is a physical procedure A logical clock is a component for catching
combined with a strategy for measuring that sequential and causal connections in a dispersed
procedure to record the progression of time. framework.
The physical clocks are based on cyclic A logical clock allows global ordering on
processes such as a events from different processes.
celestial rotation.

A FRAMEWORK FOR A SYSTEM OF LOGICAL CLOCKS

A system of logical clocks consists of a time domain T and a logical clock C. Elements of T form
a partially ordered set over a relation <. This relation is usually called the happened before or
causal precedence.

The logical clock C is a function that maps an event e in a distributed system to an element
in the time domain T denoted as C(e).

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

suc
h that for any two events ei and ej,. eiej C(ei)<
C(ej).
This monotonicity property is called the clock consistency condition.When T and C
satisfy the following condition,

Then the system of clocks is strongly consistent.

Implementing logical clocks

The two major issues in implanting logical clocks are:
Data structures: representation of each process
Protocols: rules for updating the data structures to ensure consistent conditions.

Data structures:
Each process pimaintains data structures with the given capabilities:
• A local logical clock (lci), that helps process pi measure itsown progress.
• A logical global clock (gci), that is a representation of process pi’s local view of the
logical global time. It allows this process to assignconsistent timestamps to its local
events.

Protocol:
The protocol ensures that a process’s logical clock, and thus its view of theglobal
time, is managed consistently with the following rules:
Rule 1: Decides the updates of the logical clock by a process. It controls send,
receive and other operations.
Rule 2: Decides how a process updates its global logical clock to update its view of the
global time and global progress. It dictates what information about the logical
time is piggybacked in a message and how this information is used by the receiving
process to update its view of the global time.

SCALAR TIME
Scalar time is designed by Lamport to synchronize all the events in distributed
systems. A Lamport logical clock is an incrementing counter maintained in each
process. This logical clock has meaning only in relation to messages moving between
processes. When a process receives a message, it resynchronizes its logical clock with
that sender maintainingcausal relationship.
The Lamport’s algorithm is governed using the following rules:
 The algorithm of Lamport Timestamps can be captured in a few rules:
 All the process counters start with value 0.
 A process increments its counter for each event (internal event, message
sending, message receiving) in that process.
 When a process sends a message, it includes its (incremented) counter value
with the message.

 On receiving a message, the counter of the recipient is updated to the greater of

its current counter and the timestamp in the received message, and then
incremented by one.

 If Ci is the local clock for process Pi then,

 if a and b are two successive events in Pi, then Ci(b) = Ci(a) + d1, where d1 > 0
 if a is the sending of message m by Pi, then m is assigned timestamp tm = Ci(a)

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 if b is the receipt of m by Pj, then Cj(b) = max{Cj(b), tm + d2}, where d2 > 0

Rules of Lamport’s clock

Rule 1: Ci(b) = Ci(a) + d1, where d1 > 0
Rule 2: The following actions are implemented when pi receives a message m with timestamp Cm:
a) Ci= max(Ci, Cm)
b) Execute Rule 1
c) Deliver the message

Fig 1.20: Evolution of scalar time

Basic properties of scalar time:

1. Consistency property: Scalar clock always satisfies monotonicity. A monotonic
clock only increments its timestamp and never jump.Hence it is consistent.

2. Total Reordering:Scalar clocks order the events in distributed systems.But all the
events do not follow a common identical timestamp. Hence a tie breaking mechanism
is essential to order the events. The tie breaking is done through:
 Linearly order process identifiers.
 Process with low identifier value will be given higher priority.

The term (t, i) indicates timestamp of an event, where t is its time of occurrence and i
is the identity of the process where it occurred.
The total order relation () over two events x and y with timestamp (h, i) and (k, j) is given by:

A total order is generally used to ensure liveness properties in distributed algorithms.

3. Event Counting
If event e has a timestamp h, then h−1 represents the minimum logical duration,
counted in units of events, required before producing the event e. This is called height
of the event e. h-1 events have been produced sequentially before the event e
regardless of the processes that produced these events.

4. No strong consistency

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

The scalar clocks are not strongly consistent is that the logical local clock and
logical global clock of a process are squashed into one, resulting in the loss causal
dependency information among events at different processes.

VECTOR TIME
The ordering from Lamport's clocks is not enough to guarantee that if two
events precede one another in the ordering relation they are also causally related.
Vector Clocks use a vector counter instead of an integer counter. The vector clock of a
system with N processes is a vector of N counters, one counter per process. Vector
counters have to follow the following update rules:
 Initially, all counters are zero.
 Each time a process experiences an event, it increments its own counter in the
vector by one.
 Each time a process sends a message, it includes a copy of its own
(incremented) vector in the message.
 Each time a process receives a message, it increments its own counter in the
vector by one and updates each element in its vector by taking the maximum of
the value in its own vector counter and the value in the vector in the received
message.

The time domain is represented by a set of n-dimensional non-negative integer vectors in vector
time.

Rules of Vector Time

Rule 1: Before executing an event, process pi updates its local logical time as follows:

Rule 2: Each message m is piggybacked with the vector clock vt of the sender process at sending time.
On the receipt of such a message (m,vt), process
pi executes the following sequence of actions:
update its global logical time

execute R1
deliver the message m

fig 1.21: Evolution of vector scale

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

1.17.1 Basic properties of vector time

1. Isomorphism:
 “→” induces a partial order on the set of events that are produced by a
distributed execution.
 If events x and y are timestamped as vh and vk then,


 There is an isomorphism between the set of partially ordered events produced by a
distributed computation and their vector timestamps.
 If the process at which an event occurred is known, the test to
compare two timestamps can be simplified as:

2. Strong consistency
The system of vector clocks is strongly consistent; thus, by examining the vector
timestamp of two events, we can determine if the events are causally related.

3. Event counting
If an event e has timestamp vh, vh[j] denotes the number of events executed by
process pj that causally precede e.

PHYSICAL CLOCK SYNCHRONIZATION: NEWTWORK TIME

PROTOCOL (NTP)

Centralized systems do not need clock synchronization, as they work under a

common clock. But the distributed systems do not follow common clock: each system
functions based on its own internal clock and its own notion of time.The time in
distributed systems is measured in the following contexts:
 The time of the day at which an event happened on a specific machine in the network.
 The time interval between two events that happened on different machines
in the network.
 The relative ordering of events that happened on different machines in the network.

Clock synchronization is the process of ensuring that physically distributed processors have a
common notion of time.

Due to different clocks rates, the clocks at various sites may diverge with time, and
periodically a clock synchrinization must be performed to correct this clock skew in
distributed systems. Clocks are synchronized to an accurate real-time standard like
UTC (Universal Coordinated Time). Clocks that must not only be synchronized with
each other but also have to adhere to physical time are termed physical clocks. This
degree of synchronization additionally enables to coordinate and schedule actions
between multiple computers connected to a common network.
Features of NTP :
 NTP servers have access to highly precise atomic clocks and GPU clocks
 It uses Coordinated Universal Time (UTC) to synchronize CPU clock time.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 Avoids even having a fraction of vulnerabilities in information exchange

communication.
 Provides consistent timekeeping for file servers

NTP Architecture:

• NTP uses a hierarchical system of time sources.

• At the top of the structure are highly accurate time sources – typically a GPS or
atomic clock, providing Coordinated Universal Time (UTC).
• A reference clock is known as a stratum 0 server. Stratum 1 servers are directly
linked to the reference clocks or stratum 0 servers and computers run NTP servers
that deliver the time to stratum 2 servers, and so on…..
• The stratum value is 0 to 15.
• 0 is top level contains highest accuracy & 15 is low level contains lowest accuracy

NTP Server:
NTP uses a client-server architecture; one host is configured as the NTP server and all other hosts on the
network are configured as NTP clients. Consider the following example:

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Host A is our NTP client and it is configured to use a public NTP server uk.pool.ntp.org. Host A will
periodically send an NTP request to the NTP server. The server will provide accurate data and time,
enabling system clock synchronization on Host A. NTP servers listen for NTP packets on User
Datagram Protocol (UDP) port 123. The current version is NTPv4, and it is backward compatible with
NTPv3.

Basic terminologies:

If Ca and Cb are two different clocks, then:

 Time: The time of a clock in a machine p is given by the function Cp(t),where
Cp(t)= t for a perfect clock.
 Frequency: Frequency is the rate at which a clock progresses. The frequency at
’
time t of clock CaisCa (t).
 Offset:Clock offset is the difference between the time reported by a clockand
the real time. The offset of the clock Ca is given by Ca(t)− t. Theoffset of clock
C a relative to Cb at time t ≥ 0 is given by Ca(t)- Cb(t)
 Skew: The skew of a clock is the difference in the frequencies of the
clockand the perfect clock. The skew of a clock Ca relative to clock Cb at
timet is Ca’(t)- Cb’(t).
 Drift (rate): The drift of clock Ca the second derivative of the clockvalue with
respect to time. The drift is calculated as:

Clocking Inaccuracies
Physical clocks are synchronized to an accurate real-time standard like UTC
(Universal Coordinated Time). Due to the clock inaccuracy discussed above, a timer
(clock) is said to be working within its specification if:

- maximum skew rate.

1. Offset delay estimation

A time service for the Internet - synchronizes clients to UTC Reliability from
redundant paths, scalable, authenticates time sources Architecture. Thedesign of NTP
involves a hierarchical tree of time servers with primary serverat the root synchronizes
with the UTC. The next level contains secondaryservers, which act as a backup to the
primary server. At the lowest level isthe synchronization subnet which has the clients.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

2. Clock offset and delay estimation

A source node cannot accurately estimate the local time on thetarget node due to
varying message or network delays between the nodes.This protocol employs a
very common practice of performing several trialsand chooses the trial with the

minimum delay.
Fig 1.24: Behavior of clocks

Fig a) Offset and delay estimation Fig b) Offset and delay estimation
between processes from same server between processes from different servers

Let T1, T2, T3, T4 be the values of the four mostrecent timestamps. The clocks A and
B are stable andrunning at the same speed. Let a = T1 − T3 and b = T2 − T4. If the
networkdelay difference from A to B and from B to A, called differential delay, is
small, the clock offset and roundtrip delay of B relative to A at time T4are
approximately given by the following:

Each NTP message includes the latest three timestamps T1, T2, andT3, while
T4 is determined upon arrival.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Calculate Round trip delay and clock (or) Time offset value:

Round trip delay d=(T4-T1) – (T2-T3)

= (1200-1100) – (800-850)
= 100 - (-50)
= 150

Clock (or )Timeoffset t =(T2-T1) – (T3-T4) / 2

= ((800-1100) – (850-1200)) / 2
= ((-300)+(-350)) / 2
= -650/2
=-325

MESSAGE ORDERING AND GROUP COMMUNICATION

As the distributed systems are a network of systems at various physical locations, the
coordination between them should always be preserved. The message ordering means the
order of delivering the messages to the intended recipients. The common message order
schemes are First in First out (FIFO), non FIFO, causal order and synchronous order. In case
of group communication with multicasting, the causal and total ordering scheme is followed.
It is also essential to define the behaviour of the system in case of failures. The following
are the notations that are widely used in this chapter:
 Distributed systems are denoted by a graph (N, L).
 The set of events are represented by event set {E, }
 Message is denoted as mi: send and receive events as si and ri respectively.
 Send (M) and receive (M) indicates the message M send and received.
 ab denotes a and b occurs at the same process
 The send receive pairs ={(s, r) Ei x Ejcorresponds to r}

Message Ordering Paradigms

The message orderings are

(i) non-FIFO
(ii) FIFO
(iii) causal order
(iv) synchronous order

There is always a trade-off between concurrency and ease of use and implementation.

Asynchronous Executions
An asynchronous execution (or A-execution) is an execution (E, ≺) for which the causality relation
is a partial order.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 There cannot be any causal relationship between events in asynchronous execution.

 The messages can be delivered in any order even in non FIFO.
 Though there is a physical link that delivers the messages sent on it in FIFO order due
to the physical properties of the medium, a logicallink may be formed as a composite
of physical links and multiple paths mayexist between the two end points of the
logical link.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Fig 2.1: a)non FIFO executions b) FIFO executions

FIFO executions

A FIFO execution is an A-execution in which, for all

 The logical link is non-FIFO.

 FIFO logical channels can be realistically assumed when designing distributed
algorithms since most of the transport layer protocols follow connection
oriented service.
 A FIFO logical channel can be created over a non-FIFO channel by using a
separate numbering scheme to sequence the messages on each logical channel.
 The sender assigns and appends a <sequence_num, connection_id> tuple to each
message.
 The receiver uses a buffer to order the incoming messages as per the sender’s
sequence numbers, and accepts only the “next” message in sequence.
Causally ordered (CO) executions
Definition 6.3 (Causal order (CO)): A CO execution is an A-execution in
which, for all (s, r) and (s′, r′) ∈➚, (r ∼ r′ and s ≺ s′) ⇒ r ≺ r′.
 If two send events s and s′ are related by causality ordering then their
corresponding receive events r and r′ must occur in the same order at all common
destinations.
 Figure 6.2 shows an execution that satisfies CO. s2 and s1 are related by causality
but the destinations of the corresponding messages are different. Similarly for s2
ands3.

(Fig CO executions)
 Applications of Causal order: applications that requires update to shared data, to
implement distributed shared memory, and fair resource allocation in distributed
mutual exclusion.
 Definition (causal order (CO) for implementations) If send(m1) ≺ send(m2) then
for each common destination d of messages m1 and m2, deliverd(m1) ≺ deliverd(m2) must
be satisfied.
 if m1 and m2 are sent by the same process, then property degenerates to FIFOproperty.
 In a FIFO execution, no message can be overtaken by another message between
the same (sender, receiver) pair of processes.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 In a CO execution, no message can be overtaken by a chain of messages between

the same (sender, receiver) pair of processes.
 Definition (Message order (MO)): A MO execution is an A-execution in
which, for all (s, r) and (s′, r′) ∈➚, s ≺ s′ ⇒ ￢(r′ ≺ r).
 Example: Consider any message pair, say m1 and m3 in Figure 6.2(a). s1 ≺ s3 but ￢ (r3
≺ r1) is false. Hence, the execution does not satisfy MO.

a) Not a CO execution.
 Definition (Empty-interval execution) An execution (E,≺) is an empty-interval
(EI) execution if for each pair of events (s, r) ∈ ➚, the open interval set {x ∈ E | s
≺x
≺ r} in the partial order is empty.
 Example Consider any message, say m2, in Figure 6.2(b). There does not exist any
event x such that s2 ≺ x ≺ r2. This holds for all messages in the execution. Hence,
the execution is EI.

Figure: CO Execution
Corollary: An execution (E,≺) is CO if and only if for each pair of events (s,r) ∈ ➚ and
each event e ∈ E,
• weak common past: e ≺ r ⇒ ￢(s ≺ e)
• weak common future: s ≺ e ⇒ ￢(e ≺ r).

Synchronous Execution

 When all the communication between pairs of processes uses synchronous send and
receives primitives, the resulting order is the synchronous order.
 The synchronous communication always involves a handshake between the receiver
and the sender, the handshake events may appear to be occurring instantaneously and
atomically.
 The instantaneous communication property of synchronous executions requires a
modified definition of the causality relation because for each (s, r) ∈ T, the
send event is not causally ordered before the receive event.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 The two events are viewed as being atomic and simultaneous, and neither event
precedes the other.

Fig a) Execution in an asynchronous Fig b) Equivalent synchronous

system communication

Causality in a synchronous execution: The synchronous causality relation << on E is the

smallest transitive relation that satisfies the following:
S1: If x occurs before y at the same process, then x << y.

S2: If (s, r ∈ T, then) for all x ∈ E, [(x<< s ⇐⇒ x<<r) and (s<< x ⇐⇒ r<<

x)]. S3: If x<<y and y<<z, then x<<z.

Synchronous execution: A synchronous execution or S-execution is an execution (E, <<) for

which the causality relation << is a partial order.

Timestamping a synchronous execution: An execution (E, ≺) is synchronous if and only if

there exists a mapping from E to T (scalar timestamps) such that

 for any message M, T(s(M)) = T(r(M))

 for each process Pi , if ei≺ei’, then T(ei) <
T(ei’).
ASYNCHRONOUS EXECUTION WITH SYNCHRONOUS
COMMUNICATION
 When all the communication between pairs of processes is by using synchronous
 send and receive primitives, the resulting order is synchronous order.
 A distributed program that run correctly on an asynchronous system may not be
executed by synchronous primitives. There is a possibility that the program may
deadlock, as shown by the code in Figure 6.4.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Figure A communication program for an asynchronous system deadlocks when using

synchronous primitives.

Figure 6.5 Illustrations of asynchronous executions and of crowns. (a) Crown of size 2. (b) Another crown of size 2. (c)
Crown of size 3.
Executions realizable with synchronous communication (RSC)
 In an A-execution, the messages can be made to appear instantaneous if there exists
a linear extension of the execution, such that each send event is immediately
followed by its corresponding receive event. Such an A-execution that is realized
under synchronous communication is called a realizable with synchronous
communication (RSC) execution.
Non-separated linear extension: A non-separated linear extension of (E,≺) is a linear
extension of (E,≺) such that for each pair (s, r) ∈ T, the interval { x ∈ E | s ≺ x ≺ r} is
empty.

Example:

(CO Executions)
2 2 3 3 1 1
• In the above figure: <s , r , s , r , s , r > is a linear extension that is non separated.
<s2, s1, r2, s3, r3, s1> is a linear extension that is separated.

RSC execution: An A-execution (E, ≺) is an RSC execution if and only if there

exists a non-separated linear extension of the partial order (E, ≺).

 In the non-separated linear extension, if the adjacent send event and its corresponding
receive event are viewed atomically, then that pair of events shares a common past
and a common future with each other.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Crown
Let E be an execution. A crown of size k in E is a sequence <(si, ri), i ∈{0,…, k-1}> of pairs of
corresponding send and receive events such that: s0 ≺ r1, s1 ≺ r2, sk−2 ≺ rk−1, sk−1 ≺ r0.

The crown is <(s1, r1) (s2, r2)> as we have s1 ≺ r2 and s2 ≺ r1. Cyclic dependencies
may exist in a crown. The crown criterion states that an A-computation is RSC, i.e., it can be
realized on a system with synchronous communication, if and only if it contains no crown.

Timestamp criterion for RSC execution

An execution (E, ≺) is RSC if and only if there exists a mapping from E to T (scalar
timestamps) such that

Hierarchy of ordering paradigms

The orders of executions are:

 Synchronous order (SYNC)
 Causal order (CO)
 FIFO order (FIFO)
 Non FIFO order (non-FIFO)
The Execution order have the following results
 For an A-execution, A is RSC if and only if A is an S-execution.
 RSC ⊂ CO ⊂ FIFO ⊂ A
 This hierarchy is illustrated in Figure 2.3(a), and example executions of each class are
shown side-by-side in Figure 2.3(b)
 The above hierarchy implies that some executions belonging to a class X will not
belong to any of the classes included in X. The degree of concurrency is most in A
and least in SYNC.
 A program using synchronous communication is easiest to develop and verify.
 A program using non-FIFO communication, resulting in an A execution, is hardest to
design and verify.

Fig (a) Fig (b)

Fig : Hierarchy of execution classes

Simulations
 The events in the RSC execution are scheduled as per some non-separated linear
extension, and adjacent (s, r) events in this linear extension are executed sequentially
in the synchronous system.
 The partial order of the asynchronous execution remains unchanged.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 If an A-execution is not RSC, then there is no way to schedule the events to make
them RSC, without actually altering the partial order of the given A-execution.
 However, the following indirect strategy that does not alter the partial order can be
used.
 Each channel Ci,j is modeled by a control process Pi,j that simulates the channel buffer.
 An asynchronous communication from i to j becomes a synchronous communication
from i to Pi,j followed by a synchronous communication from Pi,j to j.
 This enables the decoupling of the sender from the receiver, a feature that is essential
in asynchronous systems.

Fig : Modeling channels as processes to simulate an execution using

asynchronous primitives on synchronous system

Synchronous programs on asynchronous systems

 A (valid) S-execution can be trivially realized on an asynchronous system by
scheduling the messages in the order in which they appear in the S-execution.
 The partial order of the S-execution remains unchanged but the communication
occurs on an asynchronous system that uses asynchronous communication primitives.
 Once a message send event is scheduled, the middleware layer waits for
acknowledgment; after the ack is received, the synchronous send primitive completes.

Synchronous program order on an asynchronous system

There do not exist real systems with instantaneous communication that allows for
synchronous communication to be naturally realized.
Non-determinism
 This suggests that the distributed programs are deterministic, i.e., repeated runs of
the same program will produce the same partial order.
 In many cases, programs are non-deterministic in the following senses
1. A receive call can receive a message from any sender who has sent a message, if
the expected sender is not specified.
2. Multiple send and receive calls which are enabled at a process can be executed
in an interchangeable order.
 If i sends to j, and j sends to i concurrently using blocking synchronous calls, it
results in a deadlock.
 However, there is no semantic dependency between the send and immediately
following receive. If the receive call at one of the processes is scheduled before the
send call, then there is no deadlock.
 Rendezvous
Rendezvous (“meet with each other”)

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 One form of group communication is called multiway rendezvous, which is a

synchronous communication among an arbitrary number of asynchronous
processes.
 Rendezvous between a pair of processes at a time is called binary
rendezvous as opposed to the multiway rendezvous.
 Observations about synchronous communication under binary rendezvous:
o For the receive command, the sender must be specified eventhough the
multiple recieve commands exist.
o Send and received commands may be individually disabled or enabled.
o Synchronous communication is implemented by scheduling messages
using asynchronous communication.
 Scheduling involves pairing of matching send and receive commands that are
both enabled.
 The communication events for the control messages do not alter the partial
order of execution.
Algorithm for binary rendezvous
 Each process, has a set of tokens representing the current interactions that are
enabled locally. If multiple interactions are enabled, a process chooses one of them
and tries to “synchronize” with the partner process.
 The scheduling messages must satisfying the following constraints:
o Schedule on-line, atomically, and in a distributed manner.
o Schedule in a deadlock-free manner (i.e., crown-free), such that both the
sender and receiver are enabled for a message when it is scheduled.
o Schedule to satisfy the progress property (i.e., find a schedule within a
bounded number of steps) in addition to the safety (i.e., correctness)
property.
 Additional features of a good algorithm are:
(i) symmetry or some form of fairness, i.e., not favoring particular processes
(ii) efficiency, i.e., using as few messages as possible
 A simple algorithm by Bagrodia, makes the following assumptions:
1. Receive commands are forever enabled from all processes.
2. A send command, once enabled, remains enabled until it completes.
3. To prevent deadlock, process identifiers are used to break the crowns.
4. Each process attempts to schedule only one send event at any time.
 The algorithm illustrates how crown-free message scheduling is achieved on-line.

Messages used to implement synchronous order. Pi has higher priority than Pj . (a) Pi issues SEND(M).
(b) Pj issues SEND(M).

 The message types used are:

(i) M – Message is the one i.e., exchanged between any two process during execution
(ii) ack(M) – acknowledgment for the received message M ,

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

(iii) request(M) – when low priority process wants to send a message M to

the high priority process it issues this command.
(iv) permission(M) – response to the request(M) to low priority process from the
high priority process.

(Examples showing how to schedule messages sent with synchronous primitives)

 A cyclic wait is prevented because before sending a message M to a higher priority

process, a lower priority process requests the higher priority process for permission
to synchronize on M, in a non-blocking manner.
 While waiting for this permission, there are two possibilities:
1. If a message M′ from a higher priority process arrives, it is processed by
a receive and ack(M′) is returned.Thus, a cyclic wait is prevented.
2. Also, while waiting for this permission, if a request(M′) from a lower priority
process arrives, a permission(M′) is returned and the process blocks until M′
actually arrives.
Algorithm A simplified implementation of synchronous order.
Code shown is for process Pi , 1 ≤ i ≤ n.

(message types)
M, ack(M), request(M), permission(M)
(1) Pi wants to execute SEND(M) to a lower priority process Pj:
 Pi executes send(M) and blocks until it receives ack(M) from Pj. The send
event SEND(M) now completes.
 Any M′ message (from a higher priority processes) and request(M′) request
for synchronization (from a lower priority processes) received during the
blocking period are queued.
(2) Pi wants to execute SEND(M) to a higher priority process Pj:
(2a) Pi seeks permission from Pj by executing
send(request(M)).
(2b) While Pi is waiting for permission, it remains
unblocked.
(i) If a message M′ arrives from a higher priority process Pk, Pi accepts
M′ by scheduling a RECEIVE(M′) event and then executes send(ack(M′))
to Pk.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

(ii) If a request(M′) arrives from a lower priority process Pk, Pi executes

send(permission(M′)) to Pk and blocks waiting for the message M′.
When M′ arrives, the RECEIVE(M′) event is executed.
(2c) When the permission(M) arrives, Pi knows partner Pj is synchronized
and Pi executes send(M). The SEND(M) now completes.
(3) request(M) arrival at Pi from a lower priority process Pj:
At the time a request(M) is processed by Pi, process Pi executes
send(permission(M)) to Pj and blocks waiting for the message M. When M arrives,
the RECEIVE(M) event is executed and the process unblocks.
(4) Message M arrival at Pi from a higher priority process Pj:
At the time a message M is processed by Pi, process Pi executes
RECEIVE(M) (which is assumed to be always enabled) and then send(ack(M)) to
Pj .
(5) Processing when Pi is unblocked:
When Pi is unblocked, it dequeues the next (if any) message from the queue
and processes it as a message arrival (as per rules 3 or 4).

GROUP COMMUNICATION
Group Communication:
A group is a collection of interconnected processes with abstraction. This
abstraction is to hide the message passing so that the communication looks like a normal
procedure call.
Group communication also helps the processes from different hosts to work
together and perform operations in a synchronized manner, therefore increasing the overall
performance of the system.
Group communication is done by broadcasting of messages.

Unicast Communication :
When the host process tries to communicate with a single process in a
distributed system at the same time. A point-to-point message communication

Multicast Communication : When the host process tries to communicate with a designated group of
processes in a distributed system at the same time.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Broadcast Communication : When the host process tries to communicate with every process in a
distributed system at same time.

The network layer protocol cannot provide the following functionalities:

*Application-specific ordering semantics on the order of delivery of messages.

*Adapting groups to dynamically changing membership.
* Sending multicasts to an arbitrary set of processes at each send event.
*Providing various fault-tolerance semantics.
*The multicast algorithms can be open or closed group.

Multicasting Communication is most appropriate for group communication in DS:

 Fault tolerance based on replicated services
 Locating objects in DS
 Better performance
 Multiple updates

Differences between closed and open group algorithms:

Closed group algorithms Open group algorithms

If sender is also one of the receiver in the If sender is not a part of the communication
multicast algorithm, then it is closed group group, then it is open group algorithm.
algorithm.
They are specific and easy to implement. They are more general, difficult to design and
expensive.
It does not support large systems where client It can support large systems.
processes have short life.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

CAUSAL ORDER (CO)

In the context of group communication, there are two modes of communication:
causal order and total order. Given a system with FIFO channels, causal order needs to be
explicitly enforced by a protocol. The following two criteria must be met by a causal
ordering protocol:
 Safety: In order to prevent causal order from being violated, a message M that
arrives at a process may need to be buffered until all system wide messages sent in the
causal past of the send (M) event to that same destination have already arrived. The
arrival of a message is transparent to the application process. The delivery event
corresponds to the receive event in the execution model.
 Liveness: A message that arrives at a process must eventually be delivered to the
process.

The Raynal–Schiper–Toueg algorithm

 Each message M should carry a log of all other messages sent causally before M’s
send event, and sent to the same destination dest(M).
 The Raynal–Schiper–Toueg algorithm canonical algorithm is a representative of
several algorithms that reduces the size of the local space and message space
overhead by various techniques.
 This log can then be examined to ensure whether it is safe to deliver a message.
 All algorithms aim to reduce this log overhead, and the space and time overhead of
maintaining the log information at the processes.
 To distribute this log information, broadcast and multicast communication is used.
 The hardware-assisted or network layer protocol assisted multicast cannot efficiently
provide features:
 Application-specific ordering semantics on the order of delivery of messages.
 Adapting groups to dynamically changing membership.
 Sending multicasts to an arbitrary set of processes at each send event.
 Providing various fault-tolerance semantics

Causal Order (CO)

An optimal CO algorithm stores in local message logs and propagates on messages,
information of the form d is a destination of M about a messageM sent in the causal past, as
long as and only as long as:

Propagation Constraint I: it is not known that the message M is delivered to d.

Propagation Constraint II: it is not known that a message has been sent to d in the causal
future of Send(M), and hence it is not guaranteed using a reasoning based on transitivity that
the message M will be delivered to d in CO.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Fig : Conditions for causal ordering

The Propagation Constraints also imply that if either (I) or (II) is false, the information
“d ∈ M.Dests” must not be stored or propagated, even to remember that (I) or (II) has been
falsified:
 not in the causal future of Deliverd(M1, a)
 not in the causal future of e k, c where d ∈Mk,cDests and there is no
other message sent causally between Mi,a and Mk, c to the same
destination d.

Information about messages:

(i) not known to be delivered
(ii) not guaranteed to be delivered in CO, is explicitly tracked by the algorithm using
(source, timestamp, destination) information.
Information about messages already delivered and messages guaranteed to be delivered in
CO is implicitly tracked without storing or propagating it, and is derived from the explicit
information. The algorithm for the send and receive operations is given in Fig. 2.7 a) and b).
Procedure SND is executed atomically. Procedure RCV is executed atomically except for a
possible interruptionin line 2a where a non-blocking wait is required to meet the Delivery
Condition.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Fig a) Send algorithm by Kshemkalyani–Singhal to optimally implement causal

ordering

Fig b) Receive algorithm by Kshemkalyani–Singhal to optimally implement causal

ordering

The data structures maintained are sorted row–major and then column–major:

1. Explicit tracking:
 Tracking of (source, timestamp, destination) information for messages (i) not known to be
delivered and (ii) not guaranteed tobe delivered in CO, is done explicitly using the I.Dests
field of entries inlocal logs at nodes and o.Dests field of entries in messages.
 Sets li,aDestsand oi,a. Dests contain explicit information of destinations to which M i,ais not
guaranteed to be delivered in CO and is not known to be delivered.
 The information about d ∈Mi,a .Destsis propagated up to the earliestevents on all causal
paths from (i, a) at which it is known that M i,a isdelivered to d or is guaranteed to be
delivered to d in CO.

2. Implicit tracking:
 Tracking of messages that are either (i) already delivered, or (ii) guaranteed to be
delivered in CO, is performed implicitly.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

 The information about messages (i) already delivered or (ii) guaranteed tobe delivered
in CO is deleted and not propagated because it is redundantas far as enforcing CO is
concerned.
 It is useful in determiningwhat information that is being carried in other messages and
is being storedin logs at other nodes has become redundant and thus can be purged.
 Thesemantics are implicitly stored and propagated. This information about messages
that are (i) already delivered or (ii) guaranteed to be delivered in CO is tracked
without explicitly storing it.
 The algorithm derives it from the existing explicit information about messages (i) not
known to be delivered and (ii) not guaranteed to be delivered in CO, by examining
only oi,aDests or li,aDests, which is a part of the explicit information.

Fig : Illustration of propagation constraints

Multicasts M5,1and M4,1

Message M5,1 sent to processes P4 and P6 contains the piggybacked information M5,1.
Dest= {P4, P6}. Additionally, at the send event (5, 1), the information M 5,1.Dests = {P4,P6}
is also inserted in the local log Log 5. When M5,1 is delivered to P6, the (new) piggybacked
information P4 ∈ M5,1 .Dests is stored in Log6 as M5,1.Dests ={P4} information about P6 ∈
M5,1.Dests which was needed for routing, must not be stored in Log6 because of constraint I.
In the same way when M5,1 is delivered to process P4
at event (4, 1), only the new piggybacked information P6 ∈ M5,1 .Dests is inserted in Log4 as
M5,1.Dests =P6which is later propagated duringmulticast M4,2.

Multicast M4,3
At event (4, 3), the information P6 ∈M5,1.Dests in Log4 is propagated onmulticast M 4,3only
to process P6 to ensure causal delivery using the DeliveryCondition. The piggybacked
information on message M4,3sent to process P3must not contain this information because of
constraint II. As long as any future message sent to P6 is delivered in causal order w.r.t.
M4,3sent to P6, it will also be delivered in causal order w.r.t. M 5,1. And as M5,1 is already
delivered to P4, the information M 5,1Dests = ∅ is piggybacked on M4,3 sent to P 3. Similarly,
the information P6 ∈ M5,1Dests must be deleted from Log4 as it will no longer be needed,
because of constraint II. M5,1Dests = ∅ is stored in Log4 to remember that M5,1 has been
delivered or is guaranteed to be delivered in causal order to all its destinations.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Learning implicit information at P2 and P3

When message M4,2is received by processes P2 and P3, they insert the (new) piggybacked
information in their local logs, as information M5,1.Dests = P6. They both continue to store
this in Log2 and Log3 and propagate this information on multicasts until they learn at events
(2, 4) and (3, 2) on receipt of messages M3,3and M4,3, respectively, that any future message is
expected to be delivered in causal order to process P6, w.r.t. M5,1sent toP6. Hence by
constraint II, this information must be deleted from Log2 andLog3. The flow of events is
given by;
 When M4,3 with piggybacked information M5,1Dests = ∅ is received byP3at (3, 2),
this is inferred to be valid current implicit information aboutmulticast M 5,1because the
log Log3 already contains explicit informationP6 ∈M5,1.Dests about that multicast.
Therefore, the explicit informationin Log3 is inferred to be old and must be deleted to
achieve optimality. M5,1Dests is set to ∅ in Log3.
 The logic by which P2 learns this implicit knowledge on the arrival of M3,3is
identical.

Processing at P6
When message M5,1 is delivered to P6, only M5,1.Dests = P4 is added to Log6. Further, P6
propagates only M5,1.Dests = P4 on message M6,2, and this conveys the current implicit
information M5,1 has been delivered to P6 by its very absence in the explicit information.
 When the information P6 ∈ M5,1Dests arrives on M4,3, piggybacked as M5,1 .Dests
= P6 it is used only to ensure causal delivery of M4,3 using the Delivery Condition,
and is not inserted in Log6 (constraint I) – further, the presence of M5,1 .Dests = P4
in Log6 implies the implicit information that M5,1 has already been delivered to
P6. Also, the absence of P4 in M5,1 .Dests in the explicit piggybacked information
implies the implicit information that M5,1 has been delivered or is guaranteed to be
delivered in causal order to P4, and, therefore, M5,1. Dests is set to ∅ in Log6.
 When the information P6 ∈ M5,1 .Dests arrives on M5,2 piggybacked as M5,1. Dests
= {P4, P6} it is used only to ensure causal delivery of M4,3 using the Delivery
Condition, and is not inserted in Log6 because Log6 contains M5,1 .Dests = ∅,
which gives the implicit information that M5,1 has been delivered or is guaranteed
to be delivered in causal order to both P4 and P6.

Processing at P1
 When M2,2arrives carrying piggybacked information M5,1.Dests = P6 this (new)
information is inserted in Log1.
 When M6,2arrives with piggybacked information M5,1.Dests ={P4}, P1learns implicit
information M5,1has been delivered to P6 by the very absence of explicit information
P6 ∈ M5,1.Dests in the piggybacked information, and hence marks information P6 ∈
M5,1Dests for deletion from Log1. Simultaneously, M5,1Dests = P6 in Log1 implies
the implicit information that M5,1has been delivered or is guaranteed to be delivered in
causal order to P4.Thus, P1 also learns that the explicit piggybacked information
M5,1.Dests = P4 is outdated. M5,1.Dests in Log1 is set to ∅.
 The information “P6 ∈M5,1.Dests piggybacked on M2,3,which arrives at P 1, is
inferred to be outdated usingthe implicit knowledge derived from M5,1.Dest= ∅”
in Log1.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

TOTAL ORDER

For each pair of processes Pi and Pj and for each pair of messages Mx and My that are delivered to
both the processes, Pi is delivered Mx before My if and only if Pj is delivered Mxbefore My.

Centralized Algorithm for total ordering

Each process sends the message it wants to broadcast to a centralized process, which
relays all the messages it receives to every other process over FIFO channels.

Complexity: Each message transmission takes two message hops and exactly n
messages in a system of n processes.

Drawbacks: A centralized algorithm has a single point of failure and congestion, and is
not an elegant solution.

Three phase distributed algorithm

Three phases can be seen in both sender and receiver side.

Sender side

Phase 1
 In the first phase, a process multicasts the message M with a locally unique tag
and the local timestamp to the group members.

Phase 2
 The sender process awaits a reply from all the group members who respond with a
tentative proposal for a revised timestamp for that message M.
 The await call is non-blocking.

Phase 3
 The process multicasts the final timestamp to the group.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Fig 2.9: Sender side of three phase distributed algorithm

Receiver Side
Phase 1
 The receiver receives the message with a tentative timestamp. It updates the
variable priority that tracks the highest proposed timestamp, then revises the
proposed timestamp to the priority, and places the message with its tag and the
revised timestamp at the tail of the queue temp_Q. In the queue, the entry is marked
as undeliverable.

Phase 2
 The receiver sends the revised timestamp back to the sender. The receiver then waits
in a non-blocking manner for the final timestamp.

Phase 3
 The final timestamp is received from the multicaster. The corresponding message
entry in temp_Q is identified using the tag, and is marked as deliverable after the
revised timestamp is overwritten by the final timestamp.
 The queue is then resorted using the timestamp field of the entries as the key. As the
queue is already sorted except for the modified entry for the message under
consideration, that message entry has to be placed in its sorted position in the
queue.
 If the message entry is at the head of the temp_Q, that entry, and all consecutive
subsequent entries that are also marked as deliverable, are dequeued from temp_Q,
and enqueued in deliver_Q.

Complexity
This algorithm uses three phases, and, to send a message to n − 1 processes, it uses 3(n – 1)
messages and incurs a delay of three message hops

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

GLOBAL STATE AND SNAPSHOT RECORDING ALGORITHMS

 A distributed computing system consists of processes that do not share a common

memory and communicate asynchronously with eachother by message passing.
 Each component of has a local state. The state of the process is the local memory and
a history of its activity.
 The state of achannel is characterized by the set of messages sent along the channel
lessthe messages received along the channel. The global state of a distributedsystem is
a collection of the local states of its components.
 If shared memory were available, an up-to-date state of the entire systemwould be
available to the processes sharing the memory.
 The absence ofshared memory necessitates ways of getting a coherent and complete
view ofthe system based on the local states of individual processes.
 A meaningfulglobal snapshot can be obtained if the components of the distributed
systemrecord their local states at the same time.
 This would be possible if thelocal clocks at processes were perfectly synchronized or
if there were aglobal system clock that could be instantaneously read by the
processes.
 If processes read time froma single common clock, various indeterminatetransmission
delays during the read operation will cause the processes toidentify various physical
instants as the same time.

System Model
 The system consists of a collection of n processes, p1, p2,…,pn that are connected
by channels.
 Let Cij denote the channel from process pi to process pj.
 Processes and channels have states associated with them.
 The state of a process at any time is defined by the contents of processor registers,
stacks, local memory, etc., and may be highly dependent on the local context of
the distributed application.
 The state of channel Cij, denoted by SCij, is given by the set of messages in transit
in the channel.
 The events that may happen are: internal event, send (send (m ij)) and receive
(rec(mij)) events.
 The occurrences of events cause changes in the processstate.
 A channel is a distributed entity and its state depends on the local states of the
processes on which it is incident.

 The transit function records the state of the channel Cij.

 In the FIFO model, each channel acts as a first-in first-out message queue and,
thus, message ordering is preserved by a channel.
 In the non-FIFO model, achannel acts like a set in which the sender process adds
messages and thereceiver process removes messages from it in a random order.

A consistent global state

The global state of a distributed system is a collection of the local states ofthe
processes and the channels. The global state is given by:

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

The two conditions for global state are:

Condition 1 preserves law of conservation of messages.Condition C2 states that in the

collected global state, for everyeffect, its cause must be present.

Law of conservation of messages: Every messagem ijthat is recorded as sent in the local state of a
process pi must be capturedin the state of the channel C ij or in the collected local state of the
receiver process pj.
 In a consistent global state, every message that is recorded as received isalso recorded
as sent. Such a global state captures the notion of causalitythat a message cannot be
received if it was not sent.
 Consistent global statesare meaningful global states and inconsistent global states are
not meaningful in the sense that a distributed system can never be in an
inconsistentstate.

Issues in recording global state

The non-availability of global clock in distributed system, raises the following issues:
Issue 1:
How to distinguish between the messages to be recorded in the snapshot from those
not to be recorded?
Answer:
 Any message that is sent by a process before recording its snapshot,must be
recorded in the global snapshot (from C1).
 Any message that is sent by a process after recording its snapshot, mustnot be
recorded in the global snapshot (from C2).

Issue 2:
How to determine the instant when a process takes its snapshot?
The answer
Answer:
A process pj must record its snapshot before processing a message mij that was sent by
process pi after recording its snapshot.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Applications that use the global state information are :

 deadlocks detection
 failure recovery,
 for debugging distributed software
 If shared memory is available then an up-to-date state of the entire system is available to the
processes sharing the memory.
Example:
• Let S1 and S2 be two distinct sites of a distributed system which maintain bank accounts A and B,
respectively.
• Let the communication channels from site S1 to site S2 and from site S2 to site S1 be denoted by
C12 and C21, respectively.

Refer the diagram in PPT

Time t0: Initially, Account A=$600, Account B=$200, C12 =$0, C21=$0.
Time t1: Site S1 initiates a transfer of $50 from A to B. Hence, A= $550, B=$200, C12=$50, C21=$0.
Time t2: Site S2 initiates a transfer of $80 from Account B to A. Hence, A= $550,B=$120, C12 =$50,
C21=$80.
Time t3: Site S1 receives the message for a $80 credit to Account A. Hence, A=$630, B=$120, C12
=$50, C21 =$0.
• Suppose the local state of Account A is recorded at time t0 which is $600 and the local state of
Account B and channels C12 and C21 are recorded at time t2 are $120, $50, and $80,
respectively.
• Then the recorded global state shows $850 in the system. An extra $50 appears in the system.
• Reason: Global state recording activities of individual components must be coordinated.

SNAPSHOT ALGORITHMS FOR FIFO CHANNELS

Each distributed application has number of processes running on different physical
servers. These processes communicate with each other through messaging channels.

A snapshot captures the local states of each process along with the state of each
communication channel.
Global Snapshot = Collection of individual states of each process + individual states
of each communication channel in distributed system.
Snapshot: Is a photograph of a process taken or record it quickly

A snapshot captures the local states of each process along with the state of each communication channel.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Need for taking Snapshots

Check pointing : use a check point to restart the application incase of failure

Collecting garbage : used to remove the object that don’t have any references

Detecting deadlocks : used to examine the current application states

Debugging : To find the error

Chandy–Lamport algorithm
 The algorithm will record a global snapshot for each process channel.
 The Chandy-Lamport algorithm uses a control message, called a marker.
 Aftera site has recorded its snapshot, it sends a marker along all of its
outgoingchannels before sending out any more messages.
 Since channels are FIFO, amarker separates the messages in the channel into those to
be included in the snapshot from those not to be recorded inthe snapshot.
 This addresses issue I1. The role of markers in a FIFO systemis to act as delimiters
for the messages in the channels so that the channelstate recorded by the process at
the receiving end of the channel satisfies thecondition C2.

xample:

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Chandy-Lamport Algorithm:

Initiating a snapshot:
Process Pi initiates the snapshot
Pi records its own state and prepares a special marker message.
Send the marker message to all other processes.
Start recording all incoming messages from channels Cij for j not equal to i.

Downloaded by akileshwari Gopalakrishnan ([email protected])

 lOMoARcPSD|19961013

Propagating a snapshot
For all processes Pj consider a message on channel Ckj.
If marker message is seen for the first time:
Pjrecords own sate and marks Ckj as empty
Send the marker message to all other processes.
Record all incoming messages from channels Clj for 1 not equal to j or k.
Else add all messages from inbound channels.

Terminating a snapshot
 All processes have received a marker.
 All process have received a marker on all the N-1 incoming channels.
 A central server can gather the partial state to build a global snapshot.
Correctness of the algorithm:
 Since a process records its snapshot when it receives the first marker on any incoming channel,
no messages that follow markers on the channels incoming to it are recorded in the process’s
snapshot.
 A process stops recording the state of an incoming channel when a marker is received on that
channel.

Complexity:
The recording part of a single instance of the algorithm requires O(e) messages and O(d) time, where
e is the number of edges in the network and d is the diameter of the network.

Properties of the recorded global state :

The recorded global state may not correspond to any of the global states that occurred during the
computation.
This happens because a process can change its state asynchronously before the markers it sent are
received by other sites and the other sites record their states.
But the system could have passed through the recorded global states in some equivalent executions

 .

Downloaded by akileshwari Gopalakrishnan ([email protected])