Introduction

Distributed Systems
UON KAHONGE lecture notes
 Computer and Network
Evolution
 There has been great improvement :
 Computer Systems
 10 million dollars and 1 instruction/sec
 1000 dollars and 1 billion instructions/sec
 => a price/performance gain of 1013
 If cars had improved at this rate in the same time
period, a Rolls Royce would now cost 1 dollar and
get a 10 billion miles per gallon.(Unfortunately, it
would probably also have a 200-page manual
telling how to open the door!)
 Introduction

 Invention of high-speed computer networks

 Local-area networks (LANs)

 Small amount of information, a few microseconds

 Large amount of information, at rate of 100 million to 10

billion bits/sec

 Wide-area networks (WANs)

 64 Kbps to gigabits per second

 Consequently, it is easy to connect computer systems via

high speed networks
 DS: Definition (Tanenbaum and van Steen)
 A distributed system is a piece of
software that ensures that
 A collection of independent
computers appear to its users as a
single coherent system.
 Distributed System:
 Two aspects:
(1) independent computers and
(2) single system ⇒ middleware
 DS Definition: Lamport
 “You know you have one when the crash of
a computer you’ve never heard of stops you
from getting any work done.”
 A distributed system is one in which I
cannot get something done because a
machine I've never heard of is down.
 Application Domains
 Finance and commerce – ebay, amazon, paypal
 Information Society –www, search engines,
digital libraries
 Creative industries and entertainment – online
games(MMOG)
 Science - Grid
 Characteristics of a DS
 Multiple computers
 Concurrent execution
 Independent operation and failures
 Communications
 Ability to communicate
 No tight synchronization
 Relatively easy to expand or scale
 Transparency
 DS: Motivation
 Economics
 Share resources
 Relatively easy to expand or scale
 Speed – A distributed system may have more total
computing power then a mainframe.
 Cost
 Location Independence
 People and information are distributed
 Availability and Reliability
 If a machine crashes, the system as a whole can survive.
 Distributed System Design Goals

 Making resources available

 Distribution transparency
 Openness
 Scalability
 Distribution Transparency
 Transparency
 This is hiding the fact that its processes and
resources are physically distributed across
multiple computers
 A DS that is able to present itself to users as
if it were only on a single computer system is
said to be transparent
 Note
 Distribution transparency is a nice a goal, but
achieving it is a different story.
 Distribution Transparency
Transparency Description
Hide differences in data representation and how a resource is
Access accessed
Location Hide where a resource is located
Migration Hide that a resource may move to another location
Hide that a resource may be moved to another location while in
Relocation use

Replication Hide that a resource may be shared by several competitive users

Concurrency Hide that a resource may be shared by several competitive users

Failure Hide the failure and recovery of a resource

Persistence Hide whether a (software) resource is in memory or on disk

 Degree of Transparency
 Aiming at full distribution transparency may be too much:
 Users may be located in different continents
 Completely hiding failures of networks and nodes is
(theoretically and practically) impossible
 You cannot distinguish a slow computer from a failing one
 You can never be sure that a server actually performed an
operation before a crash
 Full transparency will cost performance, exposing
distribution of the system
 Keeping Web caches exactly up-to-date with the master
 Immediately flushing write operations to disk for fault
tolerance
 Openness of Distributed Systems
 Goal: Open distributed system -- able to interact with
services from other open systems, irrespective of the
underlying environment:
 Standard rules (protocols/interfaces) to describe
services/components
 Interfaced definitions should be:
 Complete & Vendor neutral

 These help making system / services should interoperable

& portable
 Flexibility – ability to integrate multiple components
 Achieving openness
 Achieving openness: At least make the distributed
system independent from heterogeneity of the
underlying environment:
 Hardware
 Platforms
 Languages
 Scale in Distributed Systems
 Observation: Many developers of modern
distributed system easily use the adjective
“scalable” without making clear why their
system actually scales.
 Three metrics of a scalable system:
 Number of users and/or processes (size scalability)
 Maximum distance between nodes (geographical
scalability)
 Number of administrative domains (administrative
scalability)
 Scalability Limitations
 Centralized Services: Single server for all users.
Often necessary.
 Centralized Data: Single online telephone
book.a
 Centralized Algorithms: Doing routing based
on complete information.
 Scalability Issues
 Decentralized algorithms:
 No machine has complete information about
the system state.
 Machines make decisions based only on local
information.
 Failure of one machine does not ruin the
algorithm
 There is no implicit assumption that a global
clock exists
 Scalability Issues
 Other issues with Geographical scalability
 Problems due to Synchronous communication.
 Unreliable WANs.
 Geographical scalability versus centralized
solutions.
 Scaling the system across multiple
independent administrative domains.
 Conflicting policies w.r.t to resource usage
(payment), management and security.
 Scaling Techniques
 Partition data and computations across
multiple machines
 Move computations to clients (Java applets)
 Decentralized naming services (DNS)
 Decentralized information systems (WWW)
 Make copies of data available at different
machines (Servers)
 Replicated file servers (for fault tolerance)
 Replicated databases
 Mirrored web sites
DNS Structure
 Scaling Techniques
 Allow client processes to access local copies
 Web caches (browser/Web proxy)
 File caching (at server and client)

 Hiding communication latencies

 Promoting asynchronous communication.
 Reduce overall communication for interactive
applications.
 Scaling Problems

 Applying scaling techniques is easy, except for the

following:
 Having multiple copies (cached or replicated) leads to
inconsistencies – modifying one copy makes that copy
different from the rest.
 Always keeping copies consistent requires global
synchronization.
 Global synchronization is expensive with respect to
performance.
 We have learned to tolerate some inconsistencies.
 Developing Distributed
Systems: Pitfalls
 Observation: Many distributed systems are
needlessly complex caused by mistakes that required
patching later on.
 Many possible false assumptions:
 The network is reliable
 The network is secure
 The network is homogeneous
 The topology does not change
 Latency is zero
 Developing Distributed
Systems: Pitfalls
 Bandwidth is infinite
 Transport cost is zero
 There is one administrator
 A common misconception among people when
discussing distributed systems is that it is just
another name for a network of computers. However,
this overlooks an important distinction.
 A distributed system is built on top of a network and
tries to hide the existence of multiple autonomous
computers. It appears as a single entity providing the
user with whatever services are required.
 Distributed System Challenges
 Heterogeneity – Middleware abstraction
 networks;
 computer hardware;
 operatinag systems;
 programming languages;
 implementations by different developers.
 Openness
 Scalability
 Distributed System Challenges
 Security
 Confidentiality - protection against disclosure
to unauthorized individuals
 Integrity - protection against alteration or
corruption
 Availability - protection against interference
with the means to access the resources
 Firewalls, encryption techniques
 Challenges - denial of service attacks, security
of mobile code (exes as attachments)
 Distributed System Challenges
 Failure Handling
 Detecting failures e.g. checksums for corrupt
data
 Masking failures e.g. retransmissions
 Recovery from failures
 Redundancy e.g. use of RAID
 Concurrency
 Transparency
 Quality of Service(QoS)
 Classes of Distributed Systems
 Distributed Computing Systems
 Clusters
 Grids
 Distributed Information Systems
 Transaction Processing Systems
 Eaanterprise Application Integration
 Distributed Pervasive Systems
 Home Systems
 Electronic Health systems
 Sensor Networks
 Clusters
 A cluster is a dedicated group of
interconnected computers that appears as a
single super-computer, generally used in high
performance scientific engineering and
business applications.
 In cluster computing the underlying hardware
consists of a collection of similar workstations
or PCs, closely connected by means of a high-
speed local-area network
 Clusters
 Cluster computing is used for parallel
programming in which a single (compute
intensive) program is run in parallel on multiple
machines.
 Characteristics
 Homogeneous: same OS, near-identical
hardware
 Single managing node
Cluster Computing
 Cluster Computing
 Example: Linux based Beowulf clusters
 Each cluster consists of a collection of compute
nodes that are controlled and accessed by
means of a single master node.
 The master typically handles
 Allocation of nodes to a particular parallel
program,
 Maintains a batch queue of submitted jobs,
 Provides an interface for the users of the
system.
 Cluster Computing
 The master actually runs the middleware
needed for the execution of programs and
management of the cluster, while the compute
nodes often need nothing else but a standard
operating system.
 Grids
 Grids - A grid is a type of distributed system
that enables coordinated sharing and
aggregation of distributed, autonomous,
heterogeneous resources based on users’ QoS
requirements.
 Grids are commonly used to support
applications emerging in the areas of e-Science
and e-Business.
 Grids
 This commonly involve geographically
distributed communities of people who
engage in collaborative activities to solve large
scale problems and require sharing of various
resources such as computers, data,
applications and scientific instruments.
 Characteristics
 Heterogeneous
 Dispersed across several organizations
 Can easily span a wide-area network
 Grids
 Note: To allow for collaborations, grids
generally use virtual organizations. In essence,
this is a grouping of users (or better: their IDs)
that will allow for authorization on resource
allocation.
 Given its nature, much of the software for
realizing grid computing revolves around
providing access to resources from different
administrative domains, and to only those
users and applications that belong to a specific
virtual organization.
 Grid Architecture

Layered Architecture proposed by Foster et al

 Fabric Layer
 Fabric layer provides interfaces to local
resources at a specific site.
 These interfaces are tailored to allow sharing
of resources within a virtual organization.
 They provide functions for querying the state
and capabilities of a resource, along with
functions for actual resource management
(e.g., locking resources).
 Connectivity Layer
 Connectivity layer consists of communication
protocols for supporting grid transactions that
span the usage of multiple resources.
 For exaample, protocols are needed to transfer
data between resources, or to simply access a
resource from a remote location.
 In addition, the connectivity layer will contain
security protocols to authenticate users and
resources.
 Resource Layer
 The resource layer is responsible for managing a
single resource.
 It uses the functions provided by the connectivity
layer and calls directly the interfaces made
available by the fabric layer.
 For example, this layer will offer functions for
obtaining configuration information on a specific
resource, or, in general, to perform specific
operations such as creating a process or reading
data
 Relies on the authentication performed as part of
the connectivity layer -> Access control
 Collective Layer
 It deals with handling access to multiple
resources and typically consists of services for
resource discovery, allocation and scheduling
of tasks onto multiple resources, data
replication, and so on.
 Unlike the connectivity and resource layer,
which consist of a relatively small, standard
collection of protocols, the collective layer
may consist of many different protocols for
many different purposes, reflecting the broad
spectrum of services it may offer to a virtual
organization.
 Application Layer
 The application layer consists of the
applicaations that operate within a virtual
organization and which make use of the grid
computing environment
 Grid Middleware
 The collective, connectivity, and resource layer
form the heart of what could be called a grid
middleware layer.
 These layers jointly provide access to and
management of resources that are potentially
dispersed across multiple sites.
 Transaction Processing
Systems
 Database driven applications with operations
carried out as transactions.
 Enforce ACID properties
 Enterprise Application Integration
 The more applications became decoupled from
the databases they were built upon, the more
evident it became that facilities were needed
to integrate applications independent from
their databases.
 EAI allows applications such as CRM, Supply
Management systems, business intelligence
applications to be able to share data or even
business rules
Enterprise Application Integration
 Distributed Pervasive Systems
 Distributed pervasive system is part of our
surroundings and as such, is generally
inherently distributed.
 An important feature is the general lack of
human administrative control. At best, devices
can be configured by their owners, but
otherwise they need to automatically discover
their environment and "nestle in" as best as
possible.
 Requirements for
Pervasive Applications
 Embrace contextual changes.
 Embracing contextual changes means that a
device must be continuously be aware of the
fact that its environment may change all the
time. One of the simplest changes is discovering
that a network is no longer available,
 E.g. because a user is moving between base
stations
 Requirements for
Pervasive Applications
 Encourage ad hoc composition.
 Many devices in pervasive systems will be used
in very different ways by different users. As a
result, it should be easy to configure the suite of
applications running on a device, either by the
user or through automated methods
 Requirements for
Pervasive Applications
 Recognize sharing as the default
 One very important aspect of pervasive systems
is that devices generally join the system in order
to access and possibly provide information.
 This calls for means to easily read, store,
manage, and share information. In light of the
intermittent and changing connectivity of
devices, the space where accessible information
resides will most likely change all the time.
 Distributed Pervasive Systems
 In the presence of mobility, devices should
support easy and application-dependent
adaptation to their local environment.
 They should be able to efficiently discover
services and react accordingly.
 Distribution of data, processes, and control is
inherent to these systems, for which reason it
may be better just to simply expose it rather
than trying to hide it
 Home Systems
 Systems built around home networks
 Consist of one or more personal computers,
but more importantly integrate typical
consumer electronics such as TVs, audio and
video equipment. gaming devices, (smart)
phones, PDAs, and other personal wearables
into a single sys-tem.
 Such a system should be completely self-
configuring and self-managing.
 UPnP - devices automatically obtain IP
addresses, can discover each other
 Electronic Health Care Systems
 With the increasing cost of medical treatment,
new devices are being developed to monitor
the well-being of individuals and to
automatically contact physicians when
needed.
 Personal health care systems are often
equipped with various sensors organized in a
(preferably wireless) body-area network
(BAN).
 Network should be able to operate while a
person is moving
 Electronic Health Care Systems
 Possible Organizations

 A central hub is part of the BAN and collects data as needed. From time

to time, this data is then offloaded to a larger storage device

 BAN is continuously hooked up to an external network, again through a

wireless connection, to which it sends monitored data

 Questions that arise:

 Propagation of Alerts?

 Where and how should monitored data be stored?

 How can we prevent loss of crucial data?
BAN
 Sensor Networks
 A wireless sensor network (WSN) consists of
spatially distributed autonomous sensors to
cooperatively monitor physical or
environmental conditions, such
as temperature, sound, vibration, pressure,
motion or pollutants
 Characteristics
 The nodes to which sensors are attached are:
 Many (10s-1000s)
 Simple ( hardly any memory, CPU power, or communication
facilities)
 Often battery-powered (or even battery-less)
 Sensor Network Organization
1. Sensors do not cooperate but simply send
their data to a centralized database located
at the operator's site
 waste network re-sources and energy
2. Forward queries to relevant sensors and to let
each compute an answer, requiring the
operator to sensibly aggregate the returned
answers
1. wasteful as it discards the aggregation
capabilities of sensors which would allow
much less data to be returned to the operator
 Sensor Network Organization
 Solution
 In-network processing
 Forward a query to all sensor nodes along a
tree encompassing all nodes and to
subsequently aggregate the results as they are
propagated back to the root, where the
initiator is located
 TinyDB can use any tree-based routing
algorithm