Distributed

Computing

Lecture 11:
Replication
MR.FREDRICK
BOAFO
Reference Distributed
Computing

Chapter 18
of
Coulouris G., Dollimore J., Kindberg T., Blair G., “Distributed
Systems: Concepts and Design”, 5th Edition, Addison-Wesley, 2012
Outline Distributed
Computing

• System model
• Role of group communication
• Fault-tolerant services
Introduction Distributed
Computing

• Replication involves the maintenance of copies of data on

multiple computers
• Replication provides enhanced performance, high availability
and fault tolerance
• Examples:
– Caching of web server resources in browsers and web proxies
– DNS naming service maintaining copies of name-to-attribute
mappings for computers
Motivations for replication Distributed
Computing

1) Performance enhancement:
• Example 1: Caching of data (web resources) at browsers and
proxy servers
– improves performance by means of avoiding latency of fetching the
resources from the originating server
• Example 2: Replicating the data transparently among several
originating servers in the same domain
– workload is shared between the servers by binding all the server
IP addresses to the site’s DNS name
Motivations for replication Distributed
Computing

1) Performance enhancement:
• Replication of immutable data increases performance with little
cost to the system

• Effectiveness of replication as a performance enhancement

technique is limited in case of changing data
– Require up-to-date data to be received by the client by means of
protocols designed for that purpose
Motivations for replication Distributed
Computing

2) Increased availability:
• Availability is the proportion of time for which a service is
accessible with reasonable response times
• Should be close to 100%
• Factors that affect the high availability are;
– Server failures
– Network partitions and disconnected operation
(e.g. due to user mobility)
Motivations for replication Distributed
Computing

2) Increased availability: (server failures)

• The percentage of time during which a service is available can
be enhanced by replicating the data at two or more failure-
independent servers
• Computation of availability:
Let there be n servers at which a service is replicated and
let p be the independent probability of failure for each of
these servers. Then the availability of an object (data) stored at
each of these n servers (i.e. replicated) is:

1 – probability (all managers failed or unreachable) = 1 – pn

Motivations for replication Distributed
Computing

2) Increased availability: (server failures)

• Computation of availability:
1 – probability (all managers failed or unreachable) = 1 – pn

Example:
Consider a service replicated at two servers (i.e. n = 2)
Let the probability of failing for each of these servers over a
given time period be 5% (i.e. p = 0.05)
Thus, the availability of the whole service = 1 – (0.05)2
= 1 – 0.0025
= 0.9975, i.e., 99.75%
Motivations for replication Distributed
Computing

2) Increased availability: (network partitions)

• Mobile users may either deliberately disconnect or
unintentionally get disconnected from a wireless network
Example: A user on a train with a laptop
To be able to work in such situations (called disconnected
working or disconnected operation), the user will often prepare
by copying heavily used data (e.g. contents of a shared diary)
from their usual environment to the laptop
Problem: Updates by different users may be inconsistent
(e.g. making appointment in the same slot)
• Disconnected working is feasible only when the application can
cope with stale data
Motivations for replication Distributed
Computing

3) Fault tolerance:
• Highly available data need not be strictly correct data
– E.g. two users on opposite sides of a network partition may make
updates that conflict and need to be resolved
• A fault-tolerant service, always guarantees strictly correct
behaviour despite a certain number and type of faults
• Correctness of data may depend on;
– the freshness of data and the effects of clients operations on it
– The timeliness of the service’s response
Motivations for replication Distributed
Computing

3) Fault tolerance:
• As done for high availability, replicating data and functionality
between computers, enables achieving fault tolerance.
• Estimation of required servers:
Let, there be n+1 servers providing a service. If up to n of
these crash, then at least one remains to provide the service
If up to n servers can exhibit Byzantine failures, then in
principle a group of 2n+1 servers can provide a correct
service, by having the correct servers outvote the failed
servers
• Coordination of various components must be managed
precisely by the system to maintain the correctness guarantees
Replication: Requirements Distributed
Computing

• Replication transparency
– Client should not be aware of the existence of multiple copies
• Return of only one set of values
– Client expects one result despite the fact that operations may be
performed upon more than one physical copy
• Consistency
– The operations performed upon a collection of replicated objects
produce results that meet the correctness requirements
System model Distributed
Computing
System model Distributed
Computing

• A collection of replica managers (RM) provide a service

• Data objects (e.g. bank accounts) are replicated at RMs
• Each client requests a series of operations – invocations upon
one or more of the objects.
• Operation requests may be
– read only (no updates) or update (involve reads and updates)
• Each client’s requests are first handled by a component called a
front end, which communicates with one or more of the RMs by
message passing
• Front end is the vehicle for making replication transparent
• Front end may be implemented in the client’s address space, or
as a separate process.
System model Distributed
Computing

• A single request upon replicated objects involves five phases;

1) Request
2) Coordination
3) Execution
4) Agreement
5) Response
• Actions in each phase vary according to the type of system
– E.g. a service that supports disconnected operation behaves
differently from one that provides a fault-tolerant service

• Ordering of these phases may also vary as per the type of

system
Systemmodel: Distributed
Computing
Phases of a request
1) Request:
• The front end issues the request to one or more RMs:
– either the front end communicates with a single RM, which in
turn communicates with other RMs
– or the front end multicasts the request to the RMs
2) Coordination:
• The RMs coordinate in preparation for executing the request
consistently.
• At this stage, if necessary, they agree on whether the request is
to be applied or not be applied at all if failures occur
• RMs also decide on the ordering (FIFO, Causal, Total) of the
request relative to others.
Systemmodel: Distributed
Computing
Phases of a request
2) Coordination:
• Types of ordering at RMs (same as for multicast):
– FIFO ordering: If a front end issues request r and then request r′,
any correct RM that handles r′ handles r before it
– Causal ordering: If the issue of request r happened-before issue of
request r′, any correct RM that handles r′ handles r before r′
– Total ordering: If a correct RM handles r before request r′, then
any correct RM that handles r′ handles r before it.
• Most applications require FIFO ordering.
• However, the hybrid orderings that are both FIFO and total, or
both causal and total may be used sometimes.
Systemmodel: Distributed
Computing
Phases of a request
3) Execution:
• The RMs execute the request tentatively;
– Tentative means that, they can undo its effects later
4) Agreement:
• The RMs reach consensus on the effect of the request;
– Accordingly it will be committed or aborted
5) Response:
• One or more RMs respond to the front end;
– In some systems, one replica manager sends the response
– In others, the front end receives responses from a collection of
RMs and selects or synthesizes a single response to pass back to
the client
System model: Distributed
Computing
Description and Assumptions
• The data consist of a collection of items that we call objects
• An ‘object’ could be, for example, a file or a Java object
• Each such logical object is implemented by a collection of
physical copies called replicas.
• Each of the replicas are stored at a single computer
• The data and behaviour of replica’s are tied to some degree of
consistency by the system’s operation
• The ‘replicas’ of a given object need not be identical
– Some replicas may have received updates that others have not
System model: Distributed
Computing
Description and Assumptions
• The system is asynchronous in which processes may fail only by
crashing.
• Network partitions may not occur, but, we may consider what
happens if they do occur
– Network partitions make it harder to build failure detectors
– Failure detectors are used to achieve reliable and totally ordered
multicast
• Architectural components are described by their roles
• They need not be implemented by distinct processes
System model: Distributed
Computing
Description and Assumptions
• Components called replica managers (RM) hold the replicas at a
given computer
• RM can perform operations up on replicas directly
• In a client-server environment, the RM may be a server
• Based on the application at hand, application processes can act
as both clients and replica managers
– E.g. user’s laptop on a train may contain an application that acts
as a replica manager for their diary or calendar
• An RM applies operations to its replicas recoverably
– This allows an operation at an RM does not leave inconsistent
results if it fails part way through
System model: Distributed
Computing
Description and Assumptions
• In some implementations an RM is treated as a state machine
which applies operations to its replicas atomically (indivisibly)
• A general assumption is that, each RM maintains a replica of
every object
– However, the replicas of different objects may be maintained by
different sets of replica managers.
• E.g. one object may be needed mostly by clients on one network
and another by clients on another network
• RMs may be static or dynamic
• In a dynamic system, new RMs may appear and RMs may crash,
but, such things are not allowed in a static system
The role of group communication Distributed
Computing

• Replica managers (RMs) that provide a service form a group of

processes and hence require the semantics of group
communication to manage replication effectively
• Group membership management is a crucial component to deal
with dynamic membership of RMs
– In a service that manages replicated data, for example, users may
add or withdraw a replica manager, or a replica manager may
crash and thus need to be withdrawn from the system’s operation.
The role of group communication: Distributed
Computing
Group views
• A concept called group views is employed to deal effectively
with dynamic membership
• Group views are lists of the current group members, identified
by their unique process identifiers
• The list is ordered according to the sequence in which the
members joined the group
• A new group view is generated each time that a process is
added or excluded
The role of group communication: Distributed
Computing
Suspected processes
• A group membership service may exclude a process from a
group because it is suspected, even though it may not have
crashed
– communication failure may have made the process unreachable,
while it continues to execute normally
• Effect of exclusion is that no messages will be delivered to that
process henceforth
• In the case of a closed group, if that process becomes
connected again, any messages it attempts to send will not be
delivered to the group members
• Such process will have to rejoin the group (as a ‘reincarnation’
of itself, with a new identifier), or abort its operations
The role of group communication: Distributed
Computing
Network partitions
• Disconnection or the failure of components such as a router in a
network may split a group of processes into two or more
subgroups; communication between subgroups is impossible
• Group management services deal such network partitions using
the ideas of primary partition or partitionable.
• Primary partition:
– At most one subgroup having majority to survive a partition;
remaining processes are informed to suspend operations
• Partitionable:
– Two or more subgroups continue to operate
The role of group communication: Distributed
Computing
View delivery
• For each group g the group management service delivers to any
member process p ∈ g a series of views v0(g), v1(g), v2(g), … etc.
• Example series of views:
v0(g) = (p) ---- p joins an empty group
v1(g) = (p, p′) ---- p′ joins the group
v2(g) = (p) ---- p′ leaves the group

• In the face concurrent membership changes (one process joins

the group just as another leaves), the system imposes an order
on the sequence of views given to each process.
The role of group communication: Distributed
Computing
View delivery
• Delivering a view by a member when a membership change
occurs is similar to delivering a multicast message by a process
• However, delivering a view is distinct from receiving a view
• Proposed views are held on a holdback queue until all the
existing members can agree to their delivery

• We say an event occurring in a view v(g) at process p if, at the

time of the event’s occurrence, p has delivered v(g) but has not
yet delivered the next view, v′(g)
The role of group communication: Distributed
Computing
View delivery
Basic requirements of view delivery:
• Order: If a process p delivers view v(g) and then view v′(g), then
no other process q ≠ p delivers v′(g) before v(g)
• Integrity: If process p delivers view v(g), then p ∈ v(g)
• Non-triviality: If process q joins a group and is or becomes
indefinitely reachable from process p ≠ q, then eventually q is
always in the views that p delivers.
Similarly, if the group partitions and remains partitioned, then
eventually the views delivered in any one partition will exclude
any processes in another partition.
View-synchronousgroup communication
Distributed
Computing

• Makes guarantees about the delivery ordering of view

notifications with respect to the delivery of multicast messages
• The three such guarantees are;
– Agreement: if a correct process delivers message m in view v(g),
then all other correct processes that deliver m also do so in v(g)
– Integrity: If a correct process p delivers message m, then it will
not deliver m again. Furthermore, p ∈ group(m) and the process
that sent m is in the view in which p delivers m.
– Validity (closed groups): Correct processes always deliver the
messages that they send. Let p be any correct process that delivers
message m in view v(g). If some process q ∈ v(g) does not deliver
m in view v(g), then the next view v′(g) that p delivers has q ∉
v′(g).
View-synchronousgroup Distributed
Computing
communication
Fault-tolerant services Distributed
Computing

• A service based on replication is said correct if

– it keeps responding despite failures
– clients cannot tell the difference between the service they obtain
from an implementation with replicated data and one provided by
a single correct replica manager
• Assumptions:
– communication remains reliable
– no partitions occur
– An RM (when not crashed) is assumed to behave according to a
specification of the semantics of the objects it manages
• E.g. funds transferred between bank accounts can never disappear
Fault-tolerant services Distributed
Computing

• Understanding the anomalies- a naive protocol:

Comp A Comp B Operation performed by clients

RM RM
A/C A/C A/C A/C
x y x y

Banking service

Client 1 Client 2
Fault-tolerant services Distributed
Computing

• What counts as correct behaviour for a replicated system?

• Linearizability :
– A replicated shared object service is said to be linearizable if for
any execution there is some interleaving of the series of
operations issued by all the clients that satisfies the following two
criteria:
• The interleaved sequence of operations meets the specification of
a (single) correct copy of the objects.
• The order of operations in the interleaving is consistent with the real
times at which the operations occurred in the actual execution.
– E.g. the banking service with naïve protocol in the preceding
example is not linearizable.
Fault-tolerant services Distributed
Computing

• What counts as correct behaviour for a replicated system?

• Sequential consistency :
– A replicated shared object service is said to be sequentially
consistent if for any execution there is some interleaving of the
series of operations issued by all the clients that satisfies the
following two criteria:
• The interleaved sequence of operations meets the specification of
a (single) correct copy of the objects.
• The order of operations in the interleaving is consistent with the
program order in which each individual client executedthem.
– E.g. In our banking service example, the following interleaving
satisfies sequential consistency: getBalanceA (y) → 0, getBalanceA
(x) → 0, setBalanceB(x, 1), , setBalanceA(y, 2).
Fault-tolerant services Distributed
Computing

• Passive replication (or primary-backup replication):

• There is at any one time a single primary replica manager and
one or more secondary replica managers (backups or slaves)
Fault-tolerant services Distributed
Computing

• Passive replication (or primary-backup replication):

• Sequence of events when a client requests an operation;
1) Request: Front end issues request, containing a unique ID, to the
primary RM
2) Coordination: Primary takes each request atomically, in the order
in which it receives it. Based on unique ID, if the request has
already executed, it simply resends the response
3) Execution: Primary executes the request and stores response
4) Agreement: If the request is an update, then the primary sends
the updated state, the response and the unique ID to all the
backups. The backups send an acknowledgement
5) Response: Primary responds to the front end, which hands the
response back to the client
Fault-tolerant services Distributed
Computing

• Passive replication (or primary-backup replication):

– Implements linearizability since the primary RM sequences all the
operations upon the shared objects
– If the primary RM fails, then the system retains linearizability if
• the primary is replaced by a unique backup and
• the RMs that survive agree on which operations had been
performed at the point when the replacement primary takes over
– Both of these requirements are met if the RMs are organized as a
group and if the primary uses view-synchronous group
communication to send the updates to the backups
• To survive up to f process crashes, a passive replication system
requires f +1 replica managers
• Passive replication cannot tolerate Byzantine failures)
Fault-tolerant services Distributed
Computing

• Active replication:
• The replica managers are state machines that play equivalent
roles and are organized as a group
Fault-tolerant services Distributed
Computing

• Active replication:
• Sequence of events when a client requests an operation:
1) Request: The front end attaches a unique ID to the request and
multicasts it to the group of RMS, using a totally ordered,
reliable multicast primitive
2) Coordination: The group communication system delivers the
request to every correct RM in the same (total) order
3) Execution: Every RM executes the request. Since they are state
machines and since requests are delivered in the same total
order, correct RMs all process the request identically
4) Agreement: Not needed, due to multicast delivery semantics
5) Response: Each RM sends its response to the front end
Fault-tolerant services Distributed
Computing

• Active replication:
– Achieves sequential consistency; the reliability of the multicast
ensures that every correct RM processes the same set of requests
and the total order ensures that they process them in same order
– Each front end’s requests are served in FIFO order because the
front end awaits a response before making the next request, which
is the same as ‘program order’. This ensures sequential
consistency
– The active replication system can mask up to f Byzantine failures,
as long as the service incorporates at least 2f + 1 RMs. Each front
end waits until it has collected f + 1 identical responses and
passes that response back to the client.
 Distributed
Computing

Questions and Discussion…

Thank you