Entropy, Source encoding theorem, Shannon fano coding,

Huffman coding, mutual information,channel capacity, channel coding

theorem, Error Control Coding, linear block codes, cyclic codes,
convolution codes, viterbi decoding algorithm.
 ANALOG AND DIGITAL COMMUNICATION

SOURCE CODING AND

ERROR CONTROL CODING Unit 4
4.1 INTRODUCTION TO INFORMATION THEORY

ˆˆ Information theory allow us to determine the information

content in a message signal leading to different source coding
techniques for efficient transmission of message.
ˆˆ The information theory used for mathematical modelling and
analysis of the communication system.
ˆˆ With information theory, and the modelling for communication
systems, following two main facts resolved.
i. The irreducible complexity below which the signal cannot
be compressed.
ii. The transmission rate for reliable communication of the
noisy channel.
ˆˆ In this chapter the concept of information entropy, channel
capacity, information rate etc., and source coding techniques
are discussed.
Discrete information source
ˆˆ A discrete information source which has only a finite set of
symbols is called the alphabet, and the elements of the set are
called symbols or letters.
ˆˆ A Discrete Memory less Source (DMS) can be characterized by
the list of symbols, the probability assigned to these symbols
and the specification of the rate of generating these symbols by
the source.
Uncertainty
Information is related to the probability of occurrence of event.
More is the uncertainty, more is the information associated with it.
The following example related to uncertainty (or) surprise.
Example
1. Sun rises in east
Here uncertainty is zero because there is no surprise in the
statement. The probability of occurrence of sun rising in the east is
SOURCE CODING AND ERROR CONTROL CODING

always 1
2. Sun does not rise in east:-
Hear uncertainty is high,because there is maximum information
as it is not possible

4.1.1 Definition of Information (Measure of Information)

Consider a communication system which transmits
messages m1,m2....with probabilities P1,P2,... the amount of information
transmitted through the message mkwith probability Pk is given as,
Amount of information, IK=Log2(1/PK)
Properties of information
1. If there is more uncertainty about the message, information carried
is also more
2. If receiver knows the message being transmitted,the amount of
information carried is zero
3. If I1 is information carried by message m1 and I2 is information carried
by message m2, then amount of information carried completely due
to m1and m2 is I1+I2,
4. If there m=2N equally likely messages,then amount of information
carried by each message will be N bits.

4.1.2 Concept of Amount of Information

Let us assume a communication system in which the allowable
message are m1,, m2,.....with probabilities of occurrence p1 ‘p2 .... Let the
transmitter select the message mk of probability Pk.
ˆˆ Assume that the receiver has correctly identified the message. Then
by the definition of the term information,the system has conveyed an
amount of information Ik given by
IK=log21/Pk
ˆˆ The concept of amount of information is also essential to
examine with some care the suggestion of the above equation. It can
be first noted that while Ik is an entirely dimensionless number,by
convention,the unit assigned is the bit.
ˆˆ Therefore by an example,if Pk=1/4,Ik= log24 = 2 bits. The unit bit is
employed principally as a reminder that in the above equation the
base of the logarithm is 2.(when the natural logarithmic base is used,
 ANALOG AND DIGITAL COMMUNICATION

the unit is the nat,and when the base is 10,the unit is the Hartley or
decit. The use of such unit in the present case is analogous to unit
radian used in angle measure and decibel used in connection with
power ratios.)
ˆˆ The use of base 2 is especially convenient when binary PCM is
employed ,If the 2 possible binary digit(bits) may occur with equal
likelihood,each with a probability 1/2,then the correct identification
of the binary digit conveys an amount of informational I=log22=1 bit.
ˆˆ In the past term bit was used as an abbreviation for the phrase
binary digit. When there is an uncertainly whether the word bit is
untended as an abbreviation for binary digit as binit.
ˆˆ Assume there are M equally likely and independent messages that
M=2N,with Nan integer .In this case the information in each message
is
= I log
= 2M log 2 2N = N log 2 2
ˆˆ To identify each message by binary PCM code word ,the number of
binary digits required for the each of the 2N message is also N.Hence
in this case the information in each message,as measured in bits, is
numerically the same as the number of binits needed to encode the
messages.
ˆˆ When pK=1, one possible message s allowed. In this instance,
since the receiver knows the message,there is really no need for
transmission. We find that 1= log 2, I = 0. As PK decreases from 1
to 0,Ik increases monotonically, going from to infinity. Therefore,a
greater amount of information has been conveyed when the receiver
correctly identifies a less likely message.
ˆˆ When two independent messages mK and mj are correctly identified,we
can readily prove that the amount of information conveyed is the sum
of the information associated with each of the message individually.
Therefore,we conclude that the information amount are
I k = log 2 1 / pk
I l = log 2 1 / pl

• As the messages are independent ,the probability of the composite

message is p!!! p, with corresponding information content of message
mk and mj is

I K log 2 1 / Pk Pl = log 2 1 / Pk + log 2 1 / Pl = I K + I l

SOURCE CODING AND ERROR CONTROL CODING

Problem 1
A source produces one of the four possible symbols
during each interval having probabilities p1=1/2,P2=1/4,P3=P 4=1/8.
obtain the information content of each of these symbols.

Solution
we know that the information content of each symbol is
given as, 1
I k = log 2  
 Pk 
Thus we can write
 1   1 
I 1 = log 2   = log 2   = log 2 ( 2) = 1 bit
 p1   1/ 2 
 1   1 
 = log ( 2) = 2 bits
2
I 2 = log 2   = log 2 
 p2   1/ 4 
 1   1 
 = log 2 ( 2) = 3 bits
3
I 3 = log 2   = log 2 
 p3   1/ 8 
 1   1 
 = log 2 ( 2) = 3 bits
3
I4 = log 2   = log 2 
 p4   1/ 8 

Problem 2
Calculate the amount of information,if it is given that pk=1/2.

Solution
The amount of information
 1 
I k = log 2  
 pk 
 1 
log10  
 pk  log10 2
= = = 1 bit
log10 2 log10 2
or
 1 
=log 2   = log 2 ( 2) = 1 bit
 1/ 2 
 ANALOG AND DIGITAL COMMUNICATION

Problem 3
Calculate the amount of information ,if binary digits occur with
equal likelihood in a binary pcm system.

Solution
we know that in binary PCM, there are 2 binary levels (i.e.,)1 or 0
Therefore the probabilities,
p1(0 level)=P2(1 level)=1/2
Here the amount of information content is given as,

 1 
I 1 = log 2  
 1/ 2 
 1 
I 2 = log 2  
 1/ 2 
 1  log10 2
I 1 = log 2   = log 2 ( 2) = = 1 bit
 1/ 2  log10 2
 1  log10 2
I2 = log 2   = log 2 ( 2) = = 1 bit
 1/ 2  log10 2
I1 = I 2 = 1 bit

Thus,the correct identification of the binary digits in binary PCM

carries 1 bit of information

4.2 ENTROPY OR AVERAGE INFORMATION

• In a practical communication system,it is defined as the average
information per message. Denoted ‘H’ and its units are bits/
message.
• Entropy must be as high as possible in order to ensure maximum
transfer of information.
• Thus for quantitative representation of average information per
symbol,we make the following assumptions.
I. The source is stationary so that the probabilities may remain
constant with time.
ii. Successive symbols are statistically independent and come from
the source at an average rate of r symbol per second.
m1,m2,m3.....mM and their probablities P1, P2, P3,....Pm
respectively
SOURCE CODING AND ERROR CONTROL CODING

Expression for entropy

• Consider that we have M different messages.
• Let these message be m1m2m3,...mM and their probabilities p1p2p3....
pM respectively.
• Suppose that a sequence of L message is transmitted,Then if L is very
very large then we may say that,
P1 L messages of m1 are transmitted.
P2 L messages of m2 are transmitted
PmL messages of mM are transmitted.
• The information-due to message m1 will be,

 1 
I 1 = log 2  
 p1 
• Since ,there are P1 L number of ml,the total information due to all
message of ml will be,
 1 
I 1 (total ) = P1L log 2  
 p1 

• Similarly ,the total information due to all message of m2 will be,

 1 
I 2(total ) = P1L log 2   and so on
 p2 

• Therefore ,the total information carried due to sequence of L mes-

sages will be,
I (total ) = I 1(total ) + I 2(total ) + ...Im(total )
I (total ) = p1L log 2 (1 / p1 ) + P2L log 2 (1 / p2 ) + .... + PM L log 2 (1 / pM ) ... (1)

• The average information per message will be,

Total information
Average information =
Number of messages
I (total )
=
L
• The average information is represented by entropy, which is
represented by H. Thus,

I (total )
Entropy,H=
L
 ANALOG AND DIGITAL COMMUNICATION

From equation (1), we can rewrite the above equation as,

Entropy, H = p1L log 2 (1 / p1 ) + p2L log 2 (1 / p2 ) + ... + pM L log 2 (1 / pM ) ... ( 2)

We can write the above equation using 1:(summation) sign as follows:

M
 1 
Entropy; H = ∑p k log 2  
K =1  pk 
or
M
H = − ∑ Pk log 2 pk
K =1

4.2.1 Properties of Entropy

1. Entropy (H) is zero ,if the event is sure or it is impossible
(i.e.,)H=0 if pk=0 or 1.
2.When Pk=M for all M symbols,then the symbols are equally
likely. For such a source entropy is given by,
H=log2M
3.Upper bound on entropy is given by,
Hmax ≤ log 2 M
These above properties can be proved as.
Property 1
Calculate entropy when pK=0 and when pk=1
Proof
We know that

M
 1 
H = ∑ pk log 2  
k =1  pk 
• Since,Pk=1,the above equation becomes,
M
1
H = ∑ log
K =1
 
2
1
M log10 (1)
= ∑ log ( 2)
K =1 10

= 0[sin ce log10 1=0]

• Next consider the second case,when pk=0,Instead of putting
pk=0,directly,Let us consider the limiting case,(i.e.;)
SOURCE CODING AND ERROR CONTROL CODING

M
 1 
H = ∑P K log 2  
K =1  pk 

• With pk tends to zero,the above equation will be,

M
1
H = ∑ lim P k log 2  
K =1
Pk →0
 Pk 

• The Right hand side of the above equation will be zero, when pk → 0
Hence entropy will be zero(i.e.;)
H=0
Therefore,entropy is zero for both certain and most rated message.
Property 2
When pk=1/M for all M symbols are equally likely .For such a
source entropy is given by H=log2M.
Proof
We know that the probability of M number of equally likely
messages is
1
P =
M
• This probability is same for all M messages,(i.e.,)

1
P1 = P2 = P3 = ...PM = ... (1)
M

• Entropy is given by,

M
 1 
H = ∑P K log 2  
K =1  pk 
 1   1   1 
= p1 log 2   + P2 log 2   + ...PM log 2  
 p1   p2   pM 
Putting the Probabilities form equation we get,

1 1 1
H = log 2 M + log 2 M + ... log 2 M
M M M

• In the above equation,there are M number of terms in summation.

Hence after adding these terms above equation becomes,
H=log2M
 ANALOG AND DIGITAL COMMUNICATION

Property 3
The upper bound on entropy is given as Hmax≤log2M.Hear ‘M’ is the
number of messages emitted by the source.

Solution
• To prove the above property,the property of natural logarithm is
used,it can be written as,

In x ≤ x-1 for x ≥ 0 ...(1)

• Let us consider any two probability distribution{p1,p2,...pm} and {q1,q2,...

qm} on the alphabet X={x1,x2,...xM} of the discrete memory less source.

• Consider the term

M
q 
∑P k log 2  k  .It can be written as,
K =1  Pk 

 qk 
log10  
 qk  M  Pk 
M

∑ Pk log 2  ∑ k
= P
log102
K =1  Pk  K =1

Multiply the RHS by log10e and rearrange terms as follows;

q 
log10  k 
q  log10 e  Pk 
M M

∑ pk log 2  k  = ∑ Pk 2
K =1  Pk  K =1 log10 log10 e
M
q 
= ∑ Pk log 2 e log e  k 
K =1  Pk 

 qk   qk 
Here log e   = In   Hence above equation becomes,
 Pk   Pk 

M
q  M
q 
∑P k log 2  k  = log 2 e ∑ Pk In  k 
K =1  Pk  K =1  Pk 
SOURCE CODING AND ERROR CONTROL CODING

• Form the equation(1) we can write

 qk   qk 
In  ≤ − 1
 pk   pk 

• Therefore above equation becomes,

M
 qk  M
 qk 
∑ pk log 2 
pk
 ≤ log 2
e
∑p k  − 1
K =1   K =1  pk 
M
≤ log 2e ∑ (q
K =1
k − pk )

M M

≤ log 2e  ∑ qk − ∑ pk 
 K =1 K =1 
M M
• Note that ∑q
k =1
k = 1 as well as ∑P
k =1
k =1
• Hence above equation becomes,
M
q 
∑P k log 2  k  ≤ 0
K =1  pk 

1
Now consider that qk = k for all k. That is all symbols in the alphabet
are equally likely.
Then above equation becomes,

M
 1
∑P log 2 qk + log 2  ≤ 0
k
Pk 
K =1 
M M
1
∴ ∑ Pk log 2 qk + ∑ Pk log 2 ≤0
K =1 K =1 Pk
M
1 M
∴ ∑ Pk log 2 ≤ − ∑ Pk log 2 qk
K =1 Pk K =1
M
1 1
≤ ∑P
K =1
k log 2
Pk
log 2
qk

1
Replace qk = in above equation,
M
 ANALOG AND DIGITAL COMMUNICATION

M
1 M

∑P
K =1
k log 2 ≤ ∑ Pk log 2 M
Pk K =1
M
≤ log 2 M ∑ Pk
K =1

We Know that ∑P
K =1
k =1
,hence above equation becomes,

M
1
∑P
K =1
k log 2
Pk
≤ log 2 M

The L H S of above equation is entropy H(X) with arbitrary probability

distribution.(i.e.,)

H ( X ) ≤ log 2 M

Hence proved. The maximum value of entropy is,

H max ( X ) = log 2 M

Problem 1
In binary PCM if ‘0’ occur with probability 1/4 and ‘1’ occur with
the probability equal to 3/4,then calculate the amount of information
carried by each bin it.

Solution

Here,given that binary ‘0’ has P(x1)=1/4

And binary ‘1’ has P(x2)=3/4

Then the amount of information is given as,

SOURCE CODING AND ERROR CONTROL CODING

1
I ( x i ) = log 2
P ( xi )
1
P ( x1 ) =
4
1
With P ( x1 ) =
4
log10 4
We have I ( x1 ) = log 2 4 = = 2bit
log10 2
3
And with I ( x 2 ) =
4
4
log10  
We have I ( x 2 ) =  3  = 0.415 bits
log10 2

Here,it may observed that binary ’0’ has probability 1/4 and it
carries 2 bits of information.

Whereas binary it’1’ has probability 3/4 and it carries 0.415 bits
of information.
Thus, this reveals the fact that if probability of occurrence is less,then
the information carried is more and vice versa.

Problem 2
If there are M equally likely and independent symbol,then prove
that amount of information carried by each symbol will be,
I(xi)=N bits,where M=2N and N is an integer

Solution
Since, it is given that all the M symbols are equally likely and
independent,therefore,the probability of occurrence symbol must be
1/M.
We know that amount of information is given as,
1
I ( x i ) = log 2 ... (1)
P ( xi )
 ANALOG AND DIGITAL COMMUNICATION

Here,Probability of each message is,

1
P ( xi ) =
M
Hence ,equation(1) will be,

I ( x i ) = log 2 M ... ( 2)

Further we know that M= 2N amount of information,hence equation(2)

will be

I ( x i ) = log 2 2N = N log 2 2
[Since log 22=1]
= N bits
Hence,amount of information carried by each symbol will be ‘N’
bits. We know that M=2.

This means that there ‘N’ binary digits (bin its)in each symbol.
This indicate that when the symbols are equally likely and coded with
equal number of binary digits (bin its), then the information carried by
each symbols(measured n bits) is numerically same as the number of
bin its used for each symbols.

Problem 3
Prove the statement stated as under “if a receiver knows the
message being transmitted,the amount of information carried will be
zero”.
Solution
Here it is stated that receiver “Knows” the message. This means
that only one message is transmitted. Thus,probability of occurrence o
this message will be P(xi) =1. This is because only one message and its
occurrence is certain(probability of certain events is’1’)The amount of
information carried by this type of message will be,
1 log10 1
I ( x i ) = log 2 =
P ( x i ) log10 2
Substituting(xi)=1
SOURCE CODING AND ERROR CONTROL CODING

Or
I(xi)=0 bits
This proves the statement if the receiver knows message,the
amount of information carried will be zero.
Also,as P(xi) is decreased from 1 to 0,I(xi ) increased monotonically
from 0 to infinity. This shows that the amount of information conveyed
is greater when receiver correctly identifies less likely messages.

Problem 4
Verify the following expression

I ( xixi ) =I (xi)+I ( xj ) if xi and xj are independent.

Solution
If xiand xj independent then we know that

P ( xi x j ) = P ( xi ) P ( x j )
1
also I ( x i x j ) = log 2
P ( xi x j )
1
I ( x i x j ) = log 2
P ( xi ) P ( x j )
1 1
I ( x i x j ) = log + log
P ( xi ) P (x j )
I ( xi x j ) = I ( xi ) + I ( x j )

Problem 5
A discrete source emits one of five symbols once every millisecond
with probabilities 1/2,1/4,1/8,1/16 and 1/16 respectively. Determine
the source entropy and information rate

Solution
We know that the source entropy is given as
m
1
H ( x ) = ∑ P ( X i ) log 2
i =1 P ( xi )
 ANALOG AND DIGITAL COMMUNICATION

5
1
= ∑ P ( x i ) log 2 bits / symbol
i =1 P ( xi )
1 1 1 1 1
(or ) H ( X ) = log 2 2 + log 2 4 + log 2 8 + log 2 16 + log 2 16
2 4 8 16 16
1 1 3 1 1 15
(or ) H ( X ) = + + + + =
2 2 8 4 4 8
(or ) H ( X ) = 1.875 bits/symbol
1 1
The symbol rate r= = = 1000 sym
mbols/sec
Tb 10−3
Therefore,the information rate is expressed as

R = rH ( X ) = 1000 × 1.875 bits/sec

Problem 6
The probabilities of the five possible outcomes of an experiment
are given as
1 1 1 1
P ( x1 ) = , P ( x 2 ) = , P ( x 3 ) = , P ( x 4 ) = P ( x 5 ) =
2 4 8 16
Determine the entropy and information rate if there are 16 out
comes per second.
Solution
The entropy of the system is given as

5
1
H ( X ) = ∑ P ( x i ) log 2 bits / symbol
i =1 P ( xi )
1 1 1 1 1 15
(or ) H ( X ) = log 2 2 + log 2 4 + log 2 8 + log 2 16 + log 2 16 =
2 4 8 16 16 8
H ( X ) = 1.875bits / outcome

Now, rate of information r=16 outcomes/sec.

Therefore,rate of information R will be

R = rH ( X ) = 16 × (15 / 8 ) = 30bits /sec

SOURCE CODING AND ERROR CONTROL CODING

Problem 7
An analog signal is band limited to fm Hz and sampled at Nyquist
rate. The samples are quanti zed into four levels. Each level represents
one symbol. Thus there are four symbols. The probabilities of these
four levels(symbols) are P(xi)=P(x4)=1/8 and P(x2)=P(x3)=3/8. Obtain
information rate of the source.
Solution
We are given four symbols with probabilities p(x1)=P(x4)=1/8 and
P(x2)=P(x3)=3/8. Average information H(X)(or entropy)is expressed as,

1 1 1 1
H ( X ) = P ( x1 ) log 2 + P ( x 2 ) log 2 + P ( x 3 ) log 2 + P ( x 4 ) log 2
P ( x1 ) P ( x2 ) P ( x3 ) P (x4 )

Substituting all the given values we get

1 3 8 3 8 1
H (X ) = log 2 8 + log 2 + log 2 + log 2 8
8 8 3 8 3 8
(or ) H ( X ) = 1.8bits / symbols
It is given that the signal is sampled at Nyquist rate for fm Hz band
limited signal is,
Nyquist rate =2 fm samples/sec
Since every sample generated one source symbol,
Therefore,symbols per second,r=2 fm symbols/sec
Information rate is given by : R=r H(x)
Putting values of r and H(X) in this equation,we get
R=2 fm symbols/sec X 1.8 bits/symbols
=3.6 fm bits/sec.

In this example there are four levels. Those four levels may be
coded using binary PCM as show in Table 6.1
 ANALOG AND DIGITAL COMMUNICATION

Symbol or
S.No Probability Binary digits
level
1 Q1 1/8 00
2 Q2 3/8 01
3 Q3 3/8 10
4 Q4 1/8 11
Table 6.1
Hence,two binary digits(bin its) are required to send each
symbols are sent at the rate of 2fm symbols/sec. Therefore,transmission
rate of binary digits will be binary rate=2 binary/symbol×2fm sym-
bols/sec=4 fm bin its/sec. Because one bin it is capable of conveying
one bit of information,therefore the above coding scheme is capable of
conveying 4 fm bits of information per second. But in this example, we
have obtained that we are transmitting 3.6 fm bits of information per
second.This means that the information carrying ability of binary PCM
is not completely utilized by the transmission scheme.
4.3 SOURCE CODING TO INCREASE AVERAGE INFORMATION
PER BIT

4.3.1 Source coding theorem(Shannon’s first theorem)

• Coding offers the most significant application of the information
theory
• The main purpose of the coding is to improve the efficiency of the
communication system.
• Coding is a procedure, for mapping a given set of messages or
information {m1m2,....mN} into a new set of encoded messages{ c1,,c2,...
cN} in such away that the transmission is one to one (i.e) for each
message,there is only one encoded message. This is called “Source
coding”.
• The device which performs source coding is called source encoder.
• The main problem of coding technique is the development of an
efficient source encoder.
The primary requirements are:
1. The code words produced by source encoder should be in
binary nature.
2. The source code should be unique in nature. The every code
SOURCE CODING AND ERROR CONTROL CODING

word should represent unique message.

Let there be L number of messages emitted by the source.
The probability of the kth message is pkand number of bits
assigned to this message be nk. Then the average number of bits(N) in
the code word of the message are given by,
L −1
N = ∑P N
K =0
k k ... (1)

Let Nminbe the minimum value of N. Then the coding efficiency of the
source encoder is defined as,

= N min / N k ... ( 2)

• If the coding efficiency(η)approaches unity, then the source encoder

is called efficient.
• In other words,Nmin≤ N ,and coding efficiency is maximum when
Nmin≈ N
• The value of Nmin can be determined with help of Shannon s first
theorem called source coding theorem.
Statement
Given a discrete memory less source of entropy H, the average
code word length N for any distortion less source encoding is bounded
as,
N≥H ...(3)
Note:
i. (Where ,the entropy H indicates the fundamental limit on the
average number of bits peer symbol(i.e) N this limit says tha
t average number of bits per symbol cannot be made smaller them
entropy H.
ii. (ii) Hence Nmin=H and we can write the efficiency of source encoder
from equation(2)as,

η = H/N ....(4)

4.3.2 Code Redundancy

• It is measure of redundancy of bits in the encoded message sequence.
It is given by,
Redundancy r=1- code efficiency
 ANALOG AND DIGITAL COMMUNICATION

=1-η ... (5)

NOTE;Redundancy should be as low as possible.

4.4 DATA COMPACTION

In efficient signal transmission, the redundant information should

be removed from the signal prior to transmission. This operation with no
loss of information,is ordinarily performed on a signal form. It is referred
as data
Compaction or loss less data compression.
• Basically, data compression is achieved by assigning short
description to the most frequently outcomes of source output and
longer description to the less frequent out comes.
• The various source coding schemes for data compaction are:
(i)Prefix coding.
(ii) Shannon Fanocoding.
(iii) Huffman coding.
(iv) Lempel ziv coding.
• There are two algorithms of variable length coding techniques which
is done to increase the efficiency of the source encoder. They are
1. Shannon Fano algorithm
2. Huffman coding.

4.5.SHANNON FANO CODING

Need
(i)If the probability of occurrence of all the messages are not
equally likely,then average information or entropy is reduced
and Results in information rate is reduced.
(ii) This problem can be solved by coding the messages with
different number of bits.
NOTE
(i).Shannon - Fano coding is used to encode the messages
depending upon their probabilities.
(ii).This algorithm is assigns less number off or
highly probable message and more number of bits for rarely
occurring messages.
SOURCE CODING AND ERROR CONTROL CODING

Producer

An efficient code can be obtained by the following simple producer

known as Shannon Fano coding algorithm.

Step 1:List the source symbols in order of decreasing probability.

Step 2:Partition the set into two sets that are as close to equi-probable
as possible and assign 0 to the upper set and assign 1 to the lower
set.

Step 3:Continue this process each time partitioning the sets with as
nearly probabilities as possible until further partitioning is not
possible.

Problem 1

A discrete memory less source has symbols x1,x2,x3,x4,x5 with

probabilities of 0.4,0.2,0.1,0.2,0.1 respectively. Construct a Shannon
Fano code for the source and calculate code efficiency ‘ η ’.

Solution

Step 1:Arrange the given probabilities in descending order.

Given Probabilities

P1=0.4,P2=0.2,P3=0.1,P4=0.2,P5=0.1.

Probabilities in descending order

Symbols Probabilities
x1 0.4
x2 0.2
x3 0.2
x4 0.1
x5 0.1
 ANALOG AND DIGITAL COMMUNICATION

Step 2:

The initial partitioning can be done in two ways.(i.e) we can split

as equiprobable in two methods.

Method 1:

Sym- Prob- Stage Stage Stage 3 Codeword No of bits per

bol ability 1 2 message(nk)
x1 0.4 0 0 00 2
x2 0.2 0 1 01 2
x3 0.2 1 0 10 2
x4 0.1 1 1 0 110 3
x5 0.1 1 1 1 111 3

L =1 L
N = ∑ Pknk
k =0
(or ) =∑ Pknk
K =1
5
N = ∑P n
K =1
k k

= P1n1 + P2n 2 + P3n 3 + P4n 4 + P5n 5

= ( 0.4 × 2) + ( 0.2 × 2) + ( 0.1 × 3 ) + ( 0.2 × 2) + ( 0.1 × 3 )
= 0.8 + 0.4 + 0.3 + 0.4 + 0.3
= 2.2 bits/symbol.

Method II
No of
Symbol Probability Stage 1 Stage Stage Code bits per
2 3 word message
(nk)
x1 0.4 0 0 1
x2 0.2 1 0 0 100 3
x3 0.2 1 0 1 101 3
x4 0.1 1 1 0 110 3
x5 0.1 1 1 1 111 3

Table 6.3
SOURCE CODING AND ERROR CONTROL CODING

5
N = ∑P n
K =1
k k

= P1n1 + P2n 2 + P3n 3 + P4n 4 + P5n 5

= ( 0.4 × 1) + ( 0.2 × 3 ) + ( 0.1 × 3 ) + ( 0.2 × 3 ) + ( 0.1 × 3 )
= 0.4 + 0.6 + 0.3 + 0.6 + 0.3
= 2.2 bits/symbol.

Step 3: Entropy of source (H)

M
1
H = ∑ Pk log 2  
K =1  Pk 
5
1
= ∑ Pk log 2  
K =1  Pk 
1 1 1 1 1
= P1 log 2   + p2 log 2   + P3 log 2   + P4 log 2   + P5 log 2  
 P1   p2   P3   P4   P5 
 1   1   1   1   1 
= 0.4 log 2   + 0.2 log 2   + 0.1log 2   + 0.2 log 2   + 0.1 log 2  
 0.4   0.2   0.1   0.2   0.1 
 1   1   1   1   1 
log10   log10   log10   log10   log10  
= 0.4  0.4  + 0.2  0.2  + 0.1  0.1  + 0.2  0.2  + 0.1  0.1 
log10 2 log10 2 log10 2 log10 2 log10 2
= ( 0.4 × 1.3219 ) + ( 0.2 × 2.3219 ) + ( 0.1 × 3.3219 ) + ( 0.2 × 2.3219 ) + ( 0.1 × 3.3219 )
= 0.52876 + 0.46439 + 0.33219 + 0.46439 + 0.33219
= 2.12192 bits/ssymbol.

(iii) Efficiency
H
η =
N
2.12192
= = 0.96450
2.2
0 η = 96.45 0
0 0
 ANALOG AND DIGITAL COMMUNICATION

4.6 HUFFMAN CODING

• Huffman coding assigns different number of binary digits to the
messages according to their bits/symbol probabilities of occurrence.
• Since Huffman coding,one binary,digit carries almost one bit
of information,Which is the maximum information that can be
conveyed by one digit.
Procedure
Step 1:The messages are arranged in an order of decreasing
probabilities for example x3 and x4 have lowest probabilities and
hence they are put at the bottom in the column of stage -1.
Step 2:The two messages of lowest probabilities are assigned binary ‘0’
and’1’.
Step 3:The two lowest probabilities in stage I are added and the sum
is placed in stage II,such that probabilities are in descending
order.
Step 4: Now last two probabilities are assigned 0 and 1 and they are
added. The sum of last two probabilities placed in stage III such
that probabilities are in descending order. Again’0’ and’1’ is
assigned to the last two probabilities
Step 5:This prosses continued till the last stage contains only two val-
ues. These two values are assigned digits 0 and 1 and no further
repetition required. This results in a construction of tree is know
as Huffman tree.

Step 6: Start encoding with the last stage,which consist of exactly two
ordered probabilities Assign 0 as the first digit in the code words
for all the source of symbols associated with probability,assign 1
to the second probability.

Step 7: Now go back and assign 0 and 1 to the second digit for the two
probabilities that were combined in the previous step retaining
all assignments made in that stage.

Step 8: Keep regressing this way until first column is reached

SOURCE CODING AND ERROR CONTROL CODING

Problem 1
A discrete memory less source has 6 symbols x1 ,x2,x3,x4,x5,x6
with probabilities 0.30,0.35,0.20,0.12,0.08,0.05 respectively. Construct
a huffman code and calculate its efficiency also calculate redundancy of
the code.
Solution
Code words obtained in bracket in stage. We can write the code
words for the respective probabilities,as follows

Stage I Stage
Xi Stage II Stage IV Stage V
P(xi) III
x1 0.30 0.30 0.30 0.45 0.55
x2 0.25 0.25 0.25 0.30 0.45
x3 0.20 0.20 0.25 0.25
x4 0.12 0.13 0.20
x5 0.08 0.12
x6 0.05

Number of
Message Probability Code word
bits nk
x1 0.3 00 2
x2 0.25 01 2
x3 0.2 11 2
x4 0.12 101 3
x5 0.08 1000 4
x6 0.05 1001 4

Table 6.5
(iii) To find efficiency h we have to calculate average code word length(N)
and entropy (H).
M
N = ∑P n
K =1
k k where nk is code word
6
= ∑P n
K =1
k k

= P1n1 + P2n 2 + P3n 3 + P4n 4 + P5n 5 + P6n 6

= 0.30 × 2 + 0.25 × 2 + 0.20 × 2 + 0.12 × 3 + 0.08 × 4 + 0.05 × 4
= 0.60 + 0.50 + 0.40 + 0.36 + 0.32 + 0.20
= 2.38 bits/symbol
 ANALOG AND DIGITAL COMMUNICATION

Entropy

M
1
H = ∑ Pk log 2  
K =1  Pk 
6
1
= ∑ Pk log 2  
K =1  Pk 
1 1 1 1 1 1
= P1 log 2   + P2 log 2   + P3 log 2   + P4 log 2   + P5 log 2   + P6 log 2  
 P1   P2   P3   P4   P5   P6 
 1   1   1   1   1   1 
0.30 log 2   + 0.25 log 2   + 0.20 log 2   + 0.12 log 2   + 0.08 log 2   + 0.05 log 2  
 0.30   0.25   0.20   0.12   0.08   0.05 
log 0.30 log 0.25 log 0.20 log 0.12 log 0.08 log 0.05
0.30 10 + 0.25 10 + 0.20 10 + 0.12 10 + 0.08 10 + 0.05 10
log10 2 log10 2 log10 2 log10 2 log10 2 log10 2
= 0.521 + 0.5 + 0.4643 + 0.367 + 0.2915 + 0.216
= 2.3598 bits of information/message

To obtain code efficiency(h)

H 2.3598
η= = = 0.99
N 2.38
o η = 99 o
o o
Redundancy of the code(g)

γ = 1 − γ ⇒ 1 − 0.99
= 0.01

Problem 2

A discrete memory less source X has four symbols x1,x2,x3, and x4

with probabilities 1 1 1 construct a
P ( x1 ) = , P ( x 2 ) = , P ( x 3 ) = P ( x 4 ) =
2 4 8
shannon fano code has the optimum property that ni=I(xi)and the code
efficiency is 100 o/o
SOURCE CODING AND ERROR CONTROL CODING

Solution
Given
1 1 1
P ( x1 ) = , P ( x 2 ) = , P ( x 3 ) = P ( x 4 ) = ,n (i ) = ( x i )
2 4 8
 1 
We know that,I(x i ) = log 2  
 P ( xi ) 
 
 1   1 
I ( x1 ) = log 2   ⇒ log 2  
 P ( x1 )  1
2
 
 
 1 
log10  
1
  2   log 2
=   = 10
=1
log10 2 log10 2
 
 1   1 
I ( x 2 ) = log 2  ⇒ log 2  
 P ( x )  1
 2 
4
 
 
 1 
og10 
lo 
1
4
=   = 2
log10 2
 
 1   1 
I ( x 3 ) = log 2   ⇒ log 2  =3
 P ( x3 )  1
8
 
 
 1   1 
I ( x 4 ) = log 2   ⇒ log 2  =3
 P (x4 )  1
8
 
 ANALOG AND DIGITAL COMMUNICATION

Symbol Probability Stage 1 Stage 2 Stage 3 Code No of bits per

word message(nk)
x1 1/2 0 0 1
x2 1/4 1 0 10 2
x3 1/8 1 1 0 110 3
x4 1/8 1 1 1 111 3
We know that,
entropy,
4  1 
H ( X ) = ∑ P ( x i ) log 2   (or )
i =1  P ( xi ) 
4
= ∑ P ( xi ) I ( xi )
i =1

= P ( x1 ) I ( x1 ) + P ( x 2 ) I ( x 2 ) + P ( x 3 ) I ( x 3 ) + P ( x 4 ) I ( x 4 )
1  1  1  1 
=  × 1 +  × 2  +  × 3  +  × 3 
2  4  8  8 
1 1 3 3
= + + +
2 2 8 8
= 1.75 bits/message

Average code word length(N)

M M
N = ∑ P n (or )∑ P ( x ) n
K =1
k k
i =1
i i

= P ( x1 ) n1 + P ( x 2 ) n 2 + P ( x 3 ) n 3 + P ( x 4 ) n 4
1  1  1  1 
=  × 1 +  × 2  +  × 3  +  × 3 
 2   4   8   8 
= 1.75 bits/syymbol
code efficiency
H (X ) 1.75
η= = =1
N 1.75
o η = 100 o
o o
SOURCE CODING AND ERROR CONTROL CODING

Problem 3
A DMS has five equaly likely symbols. Construct a Shannon
fano code for x and calculate the efficiency of code. Construct another
Shannon- fano code and compare the results. Repeat for the Huffman
code and compare results.

Solution

(i)A Shannon fano code[by choosing two approximately equi-

probable (0.4 versus 0.6) sets] is constructed as follows

Symbol Probability Stage

1
Stage Stage Code
2 3 word
No of bits per
message(nk)
x1 0.2 0 0 00 2
x2 0.2 0 1 01 2
x3 0.2 1 0 10 2
x4 0.2 1 1 0 110 3
x5 0.2 1 1 1 111 3

Entropy

5  1 
H ( X ) = ∑ P ( x i ) log 2  
i =1  P ( xi ) 
Here all five probabilitie es are same(i.e.,) 0.2 so we can write,
 1 
H ( X ) = 5 × P ( x i ) log 2  
 P ( xi ) 
 1 
= 5 × 0.2 × log 2  
 0.2 
 1 
0.2 log10  
= 5×  0.2 
log10 ( 2)
H ( X ) = 2.32 bits/message
 ANALOG AND DIGITAL COMMUNICATION

Average code word length N ( )

5
N = ∑ Pk nk
k =1

= ( 0.2 × 2) + ( 0.2 × 2) + ( 0.2 × 2) + ( 0.2 × 3 ) + ( 0.2 × 3 )

= 0.4 + 0.4 + 0.4 + 0.6 + 0.6
= 2.4 bits/symbol.
H (X )
coding efficiency η=
N
2.32
= = 0.967
2.4
o η = 96.7 o
o o

(ii) Another method for Shannon fano code[by choosing another two
approximately equiprobable (0.6 versus 0.4) sets]is constructed as
follows
Stage Stage Stage Cord No of bits per
Symbol Probability
1 2 3 word message (nk)
x1 0.2 0 0 00 2
x2 0.2 0 1 0 010 3
x3 0.2 0 1 1 011 3
x4 0.2 1 0 10 2
x5 0.2 1 1 11 2

The entropy H(X) is same as in the previous method,(i.e.,)

5  1 
H ( X ) = ∑ P ( x i ) log 2  
i =1  P ( xi ) 
= 2.32 bits/message
SOURCE CODING AND ERROR CONTROL CODING

Average code word length N ( )

5 5
N = ∑ P n (or ) ∑ P ( x ) n
K =1
k k
i =1
i i

= P ( x1 ) n1 + P ( x 2 ) n 2 + P ( x 3 ) n 3 + P ( x 4 ) n 4 + P ( x 5 ) n 5
= ( 0.2 × 2) + ( 0.2 × 3 ) + ( 0.2 × 3 ) + ( 0.2 × 2) + ( 0.2 × 2)
= 0.4 + 0.6 + 0.6 + 0.4 + 0.4
= 2.4 bits/symbol
ding coefficiency ( η)
Cod
H (X )
coding efficiency η=
N
2.32
= = 0.967
2.4
o η = 96.7 o
o o

Since, average code word length is same as that for the code of
part(i), the efficiency is same.
(iii)The huffman code is constructed as follows

Stage 1
xi Stage II Stage III Stage IV
P(xi)
x1 0.2 (01) 0.4 (1) 0.4 (1) 0.6 (0)
x2 0.2 (000) 0.2 (01) 0.4 (00) 0.4 (1)
x3 0.2 (001) 0.2 (000) 0.2 (01)
x4 0.2 (10) 0.2 (001)
x5 0.2 (11)

Symbol Probability Code word Length

x1 0.2 01 2
x2 0.2 000 3
x3 0.2 001 3
x4 0.2 10 2
x5 0.2 11 2

The average code word length

 ANALOG AND DIGITAL COMMUNICATION
M 5
N = ∑ P n (or ) N = ∑ P ( x ) n
K =1
k k
i =1
i i

= P ( x1 ) n1 + P ( x 2 ) n 2 + P ( x 3 ) n 3 + P ( x 4 ) n 4 + P ( x 5 ) n 5
Here all probability have same value (0.2).
=0.2 × [n1 + n 2 + n 3 + n 4 + n 5 ]
so,=
= 0.2[2 + 3 + 3 + 2 + 2]
= 0.2 × 12
= 2.4 bits/symbol
Entropy & efficiency are also same as that the Shannon fano code
due to same code word length.
Entropy
5  1 
H ( X ) = ∑ P ( x i ) log 2  
i =1  P ( xi ) 
 1   1   1 
= P ( x1 ) log 2   + P ( x 2 ) log 2   + P ( x 3 ) log 2  
 P ( x1 )   P ( x2 )   P ( x3 ) 
 1   1 
+P ( x 4 ) log 2   + P ( x 5 ) log 2  
 P (x4 )   P ( x5 ) 
Here all five probab bilities have same value as 0.2 so we can write,
 1 
=5 × P ( x1 ) lo
og 2  
 P ( x1 ) 
 1 
= 5 × 0.2 log 2  
 0.2 
 1 
0.2 log10  
= 5×  0.2 
log10 2
= 2.32 bits/message
Coding efficiency ( η)
H (X )
Coding efficiiency η =
N
2.32
= = 0.967
2.4
o η = 96.7 o
o o

Problem 4
A Discrete memory less source (DMS) has five symbols x1x2,x3,x4,
SOURCE CODING AND ERROR CONTROL CODING

and x5 with P(x1)=0.4,P(x2)=0.19,P(x3)=0.16,P(x4)=0.15,P(x5)=0.1.

(i) Construct the Shannon-fano code for x and calculate efficiency of the
code.
(ii) Repeat for the Huffman code and compare the results.

Symbol Probability Stage 1 Stage 2 Stage 3 Code No of bits per

word message (nk)
x1 0.4 0 0 00 2
x2 0.19 0 1 01 2
x3 0.16 1 0 10 2
x4 0.15 1 1 0 110 3
x5 0.1 1 1 1 111 3

Entropy

5  1 
H ( X ) = ∑ P ( x i ) log 2  
i =1  P ( xi ) 
 1   1   1 
= P ( x1 ) log 2   + P ( x 2 ) log 2   + P ( x 3 ) log 2  
 P ( x1 )   P ( x2 )   P ( x3 ) 
 1   1 
+P ( x 4 ) log 2   + P ( x 5 ) log 2  
 P (x4 )   P ( x5 ) 
 1   1   1 
= 0.4 log 2   + 0.19 log 2   + 0.16 log 2  
 0.4   0.19   0.16 
 1   1 
+0.15 log 2   + 0.1 log 2  
 0 .15   0.1 
H ( X ) = 2.15 bits/symbol
Code efficiency ( η)
H (X ) 2.15
η= = = 0.956
N 2.25
o η = 95.6 o
o o

(ii) Huffman code

Huffman code is constructed as follows
 ANALOG AND DIGITAL COMMUNICATION

Stage I
Xi Stage II Stage III Stage IV
P(xi)
x1 0.4 (1) 0.4 (1) 0.4 (1) 0.6 (0)
x2 0.19 (000) 0.25 (01) 0.35 (00) 0.4 (1)
x3 0.16 (001) 0.19 (000) 0.25 (01)
x4 0.15 (010) 0.16 (001)
x5 0.1 (011)

Entropy H(X)
Entropy H(X) of Huffman code is same as that for the Shannon-
Fano code.

5  1 
H ( X ) = ∑ P ( x i ) log 2  
i =1  P ( xi ) 
 1   1   1 
= P ( x1 ) log 2   + P ( x 2 ) log 2   + P ( x 3 ) log 2  
 P ( x1 )   P ( x2 )   P ( x3 ) 
 1   1 
+P ( x 4 ) log 2   + P ( x 5 ) log 2  
 P (x4 )   p ( x5 ) 
 1   1   1 
= 0.4 log 2   + 0.19 log 2   + 0.16 log 2  
 0.4   0.19   0.16 
 1   1 
+0.15 log 2   + 0.1 log 2  
 0.15   0.1 
H ( X ) = 2.15 bits/message
SOURCE CODING AND ERROR CONTROL CODING

Symbol P(X) Code word Length(nk)

x1 0.4 1 1
x2 0.19 000 3
x3 0.16 001 3
x4 0.15 010 3
x5 0.1 011 3

Average code word length N ( )

5
N = ∑ P ( xi ) ni
i =1

= P ( x1 ) n1 + P ( x 2 ) n 2 + P ( x 3 ) n 3 + P ( x 4 ) n 4 + P ( x 5 ) n 5
= ( 0.4 × 1) + ( 0.19 × 3 ) + ( 0.16 × 3 ) + ( 0.15 × 3 ) + ( 0.1 × 3 )
N = 2.2 bits/symbol
Code efficiency ( η)
H 2.15
η= = = 0.977
N 2.2
o η = 97.7 o
o o
 ANALOG AND DIGITAL COMMUNICATION

their probabilities given

x1 x2 x3 x4 x5 x6 x7
0.05 0.15 0.2 0.05 0.15 0.3 0.1

Solution
Arranging the symbols in decreasing order and obtain the
Huffman code as follows
Stage I Stage
Xi Stage III Stage IV Stage V Stage VI
P(Xi) II
x6 0.3 (00) 0.3 (00) 0.3 (00) 0.3 (00) 0.4 (1) 0.6 (0)
x3 0.2 (10) 0.2 (10) 0.2 (10) 0.3 (10) 0.3 (00) 0.4 (1)
x2 0.15 (010) 0.15 (010) 0.2 (11) 0.2 (10) 0.3 (01)
x5 0.15 (011) 0.15 (011) 0.15 (010) 0.2 (11)
x7 0.1 (110) 0.1 (110) 0.15 (011)
x1 0.05 (1110) 0.1 (111)
x4 0.05 (1111)

Message Probability Code Length

x1 0.05 1110 4
x2 0.15 010 3
x3 0.2 10 2
x4 0.05 1111 4
x5 0.15 011 3
x6 0.3 00 2
x7 0.1 110 3
Table 6.15
Average codeword lenght N ( )
7
N = ∑ P ( xi ) ni
i =1

= P ( x1 ) n1 + P ( x 2 ) n 2 + P ( x 3 ) n 3 + P ( x 4 ) n 4 + P ( x 5 ) n 5
+ P ( x6 )n6 + P ( x7 )n7
= ( 0.05 × 4 ) + ( 0.15 × 3 ) + ( 0.2 × 2) + ( 0.05 × 4 ) + ( 0.15 × 3 )
+ ( 0.3 × 2) + ( 0.1 × 3 )
N = 2.6 bits/symboll
Entropy H(X)
SOURCE CODING AND ERROR CONTROL CODING
7  1 
H ( X ) = ∑ P ( x i ) log 2  
i =1  P ( xi ) 
 1   1   1   1 
= 0.05 log 2   + 0.15 log 2   + 0.2 log 2   + 0.05 log 2  
 0.05   0.15   0.2   0.05 
 1   1   1 
+0.15 log 2   + 0.3 log 2   + 0.1 log 2  
 0.15   0.3   0.1 
H ( X ) = 2.57 bits/message
Coding efficiency ( η )
H 2.57
η= = = 0.9885
N 2.6
o η = 98.85 o
o o

Problem 7
A discrete memory less source has a alphabet given below.
Compute two different Huffman codes for this source,hence for each of
the two codes,find,
(i) The average code-word length.
(ii) The variance of the average code-word length over the
ensemble of source symbol.

Symbol S0 S1 S2 S3 S4
Probability 0.55 015 0.15 0.10 0.05

Solution
The two different Huffman codes are obtained by placing the com-
bined probability as high as possible or as low as possible.
1. Placing combined probability as high as possible

Stage Stage Stage

Symbol Stage III
I P(xi) II IV
0.55
S0 0.55 (0) 0.55 (0) 0.55 (0)
(0)
0.45
S1 0.15 (100) 0.15 (11) 0.3 (10)
(1)
S2 0.15 (101) 0.15 (100) 0.15 (11)
S3 0.1 (110) 0.15 (101)
S4 0.05 (111)
 ANALOG AND DIGITAL COMMUNICATION

Symbol Probability Code word hk

s0 0.55 0 1
s1 0.15 100 3
s2 0.15 101 3
s3 0.1 110 3
s4 0.05 111 3
1. Average code-word length
4
∴N = ∑P n
k =0
k k

= ( 0.55 × 1) + ( 0.15 × 3 ) + ( 0.15 × 3 ) + ( 0.1 × 3 ) + ( 0.05 × 3 )

= 1.9 bits/symbol
(ii )Variance of the code
4 2
σ2 = ∑P
k =0
k
nk − N 
 

= 0.55 [1 − 1.9] + 0.15 [3 − 1.9] + 0.15 [3 − 1.9] + 0.1[3 − 1.9]

2 2 2 2

+ 0.05 [3 − 1.9]
2

= 0.99
2. Placing combined probability as low as possible
Stage I Stage Stage
Symbol Stage IV
P(Xi) II III
s0 0.55 (0) 0.55 (0) 0.55 (0) 0.55 (0)
s1 0.15 (11) 0.15 (11) 0.3 (10) 0.45 (1)
s2 0.15 (100) 0.15 (100) 0.15 (11)
s3 0.1 (1010) 0.15 (101)
s4 0.05 (1011)
(i ) Average code-word length
4
∴N = ∑P n
K =0
k k

= ( 0.55 × 1) + ( 0.15 × 2) + ( 0.15 × 3 ) + ( 0.1 × 4 ) + ( 0.05 × 4 )

= 1.9 bits/symbol
(ii )Variance of the code
4 2
σ2 = ∑P
K =0
k
nk − N 
 
SOURCE CODING AND ERROR CONTROL CODING

= 0.55 [1 − 1.9] + 0.15 [2 − 1.9] + 0.15 [3 − 1.9] + 0.1[ 4 − 1.9]

2 2 2 2

+ 0.05 [4 − 1.9]
2

= 1.29

Average code-word
Method Varaiance
length
As high as possible 1.9 0.99
As low as possible 1.9 1.29

4.7 MUTUAL INFORMATION

Mutual information I(X,Y) of a channel is defined as amount

of information transferred when xi transferred and transmitted and yj
received .Its represented by,I(xi,yj)

x 
P i 
 yj 
I ( x i , y j ) = log   bits ... (1)
P ( xi )

 xi 
P
Here I(xi,yj) is the mutual information,  y  is the conditional
 j 

Probability that xi was transmitted and yj is received. P(xi) is the

probability of symbol xi for transmission.
The average mutual information is represented by I(X;Y).It is
calculated in bits/symbol. Average mutual information is defined as the
amount of source information gained per received symbol. It is given by
m n
I ( X ;Y ) = ∑ ∑ P ( x i , y j ) I ( x i y j ) ... ( 2)
i =1 j =1

Substitue (1) in (2)

  x 
P  i 
m m
  yj  
I ( X ;Y ) = ∑ ∑ P ( x i , y j ) log 2    
i =1 j =1
 P ( xi ) 
 
 
 ANALOG AND DIGITAL COMMUNICATION

Properties of Mutual information

(i) The mutual information of a channel is symmetric.

I ( X ;Y ) = I (Y ; X )
(ii) The mutual information can be expressed in terms in terms of
entropies of channel input or channel out put and conditional entropies.

I ( X ;Y ) = H ( X ) − H ( X /Y )
I (Y ; X ) = H (Y ) − H (Y / X )
where , H (X /Y ) and H (Y / X ) are conditional entropies.

(iii) The mutual information is always positive.

I ( X ;Y ) ≥ 0

(iv) The mutual information is related to the joint entropy H(X,Y) by

following relation,

I ( X ;Y ) = H ( X ) + H (Y ) − H ( X ,Y )

Property 1
The mutual information of a channel is symmetric.
(i.e.,) I(X;Y)=I(Y;X)
Proof
Let us consider some standard relationships from probability
theory.These are as follows
X 
P ( X i ,Y j ) = P  i  P (Y j ) ... (1)
 Yj 
 
 Yj 
and P ( X i ,Y j ) = P   P ( Xi ) ... ( 2)
 Xi 
From equation (1) and (2) we can write,
X   Yj 
P i
 Yj  P (Y j ) = P   P ( Xi ) ... ( 3 )
   Xi 

Therefore,the average mutual information is given by

SOURCE CODING AND ERROR CONTROL CODING

 X 
P  i 
m n
  Y j  
I ( X ;Y ) = ∑ ∑ P ( X i ,Y j ) log 2   
i =1 j =1
 P ( Xi ) 
 
 

Hence we can write I(X;Y) as follows;

  Yj  
P  
m n X
I ( X ;Y ) = ∑ ∑ P ( X i ,Y j ) log 2   i  
i =1 j =1
 P (Y ) 
 j

 

By considering equation (3) the above equation can be written as,

 X 
P  i 
m n
  Y j  
I (Y ; X ) = ∑ ∑ P ( X i ,Y j ) log 2   
i =1 j =1
 P ( Xi ) 
 
 
= I ( X ;Y )

Thus the mutual information of the discrete memory less channel is

symmetric
Property 2
I (X;Y)=H (X)-H (X/Y)
I (Y;X)=H (Y)-H (Y/X)
Solution H(X/Y) is the conditional entropy and it is given as,

m n
1
H ( X /Y ) = ∑ ∑ P ( X i ,Y j ) log 2 ... (1)
i =1 j =1 P ( X i /Y j )

H(X/Y) is the information or uncertainly in X after Y is received. In

other words H(X/Y) is the information lost in the noisy channel. It is the
average conditional self information.

We know that,average mutual information is given as,

 ANALOG AND DIGITAL COMMUNICATION

 X 
P  i 
m n
  Y j  
I ( X ;Y ) = ∑ ∑ P ( X i ,Y j ) log 2   
i =1 j =1
 P ( Xi ) 
 
 

Let us write the above equation as,

m n
1
I ( X ;Y ) = ∑ ∑ P ( X i ,Y j ) log 2
i =1 j =1 P ( Xi )
m n
1
−∑ ∑ P ( X i ,Y j ) log 2
i =1 j =1 P ( X i ,Y j )

From equation(1) above equation can be rewritten as,

m n
1
I ( X ;Y ) = ∑ ∑ P ( X i ,Y j ) log 2 − H ( X /Y ) ... ( 2)
i =1 j =1 P ( Xi )

We know that the standard probability relation which is given as follows

∑ P ( X ,Y ) = P ( X )
j =1
i j i

Hence equation (2) will be,

m
1
I ( X ;Y ) = ∑ P ( X i ) log 2 − H ( X /Y ) ... ( 3 )
i =1 P ( Xi )
First term of the above equatiion represents entropy.(i.e.,)
m
1
H ( X ) = ∑ P ( X i ) log 2 ... ( 4 )
i =1 P ( Xi )
Hence equation(3) becomes,
I ( X ;Y ) = H ( X ) − H ( X /Y ) ... ( 5 )

Here note that I(X;Y) is the average information transferred

per symbol across the channel. It is equal to source entropy minus
information lost in the noisy channel is given by above equation.
SOURCE CODING AND ERROR CONTROL CODING

  Yj  
P  
m n Xi  
I (Y ; X ) = ∑ ∑ P ( X i ,Y j ) log 2  
i =1 j =1
 P (Y ) 
 j

 
m n
1
= ∑ ∑ P ( X i ,Y j ) log 2
j =1 i =1 P (Y j )
m n
1
−∑ ∑ P ( X i ,Y j ) log 2 ... ( 6 )
i =1 j =1 P (Y j / X i )
The conditional entropy H(Y/X) is given as,
m n
1
/X)=∑ ∑ P ( X i ,Y j ) log 2
H(Y/ ... ( 7 )
i =1 j =1 P (Y j / X i )

Here H (Y/X) is the uncertainty in Y when X was transmitted.

With this result, equation (6) becomes,
m n
1
I (Y ; X ) = ∑ ∑ P ( X i ,Y j ) log 2 − H (Y / X ) ... ( 8 )
j =1 i =1 P (Y j )
By using th
he standard probability equation,
m

∑ P ( X ,Y ) = P (Y )
i =1
i j j ... ( 9 )

Hence equation(8) becomes,

n
1
I (Y ; X ) = ∑ P (Y j ) log 2 − H (Y / X )
j =1 P (Y j )

1 n
H (Y ) = ∑ P (Y j ) log 2
We know that j =1 P (Y j )
Hence first term of above equation represents H (Y).Hence above
equation becomes,

I (Y ; X ) = H (Y ) − H (Y / X ) ... (10 )

Property 3
I(X;Y) ≥ 0
Solution
Average mutual information can be written as,
m n  P (X ) 
I ( X ;Y ) = ∑ ∑ P ( X i ,Y j ) log 2  i
 ... (1)
 P ( X i /Y j ) 
i =1 j =1
 
 ANALOG AND DIGITAL COMMUNICATION

Using Bayes rules,we have

since log 2P = log 2 e × log 2 p

 P ( X )   P ( X i ) P (Y j ) 
 i
=  log e P = In p
 P ( X i /Y j )   P ( X i ,Y j ) 
    1
In e=
In 2
We can write equation (1)) as under

1 m n  P ( X i ) P (Y j ) 
-I (Y ; X ) = P ( X ,Y )
∑ ∑ i j  P X ,Y In ... ( 2)
In 2 i =1 j =1
 ( i j) 
Also we know that
In α ≤ α − 1
There fore we have

1 m n  P ( X i ) P (Y j ) 
-I (Y ; X ) ≤ P (
∑ ∑ i j  P X ,Y
X ,Y )  − 1 
In 2 i =1 j =1
 ( i j ) 

1 m n m n 
−I (Y ; X ) ≤  ∑ ∑ P ( X i ) P (Y j ) − ∑ ∑ P ( X i ,Y j )  ... ( 3 )
In 2  i =1 j =1 i =1 j =1 
Since

m n m n

∑ ∑ P ( X i ) P (Y j ) = ∑ P ( X i )∑ P (Y j ) = (1)(1)
i =1 j =1 i =1 j =1

m n m  n  m

∑ ∑ P ( X ,Y ) = ∑ ∑ P ( X ,Y ) = ∑ P ( X ) = 1
i j i j i
i =1 j =1  j =1 i =1  i =1

Equation (3)) reduces to

-I ( X ;Y ) ≤ 0
I ( X ;Y ) ≥ 0

Hence proved.
Property 4
I(X;Y) =H (X) +H (Y)-H (X,Y)
Solution
We know the relation
H (X,Y) = H (X,Y) -H (Y)
Therefore
SOURCE CODING AND ERROR CONTROL CODING

H (X/Y) = H (X,Y) -H (Y) ...(1)

Mutual information is given by
I (X;Y) =H (X) -H (X/Y) ...(2)
Substituting equation (1) in (2)
I (X;Y) =H (X) + H (Y) - H (X,Y)
Thus the required relation is proved.
Problem 1
Verify the following expression

H (X,Y) =H (X/Y) +H (Y)

Solution
We know that
P (Xi,Yj)=P (Xi/Yj) P (Yj) and
m

∑P (X
i =1
i /Y j ) = P (Y j )

Also ,we have

n m
H ( X ,Y ) = −∑ ∑ P ( X i ,Y j ) log P ( X i ,Y j )
j =1 i =1
n m
H ( X ,Y ) = −∑ ∑ P ( X i ,Y j ) log P ( X i /Y j ) P (Y j ) 
j =1 i =1
n m
H ( X ,Y ) = −∑ ∑ P ( X i ,Y j ) log P ( X i /Y j ) 
j =1 i =1

n
 m

−∑  ∑ P ( X i ,Y j )  log P (Y j )
j =1  i =1 

4.8 CHANNEL CAPACITY

• The mutual information I(X;Y) represented a measure of the average

information per symbol transmitted in the system.
• A suitable measure for efficiency of transmission of information may
be introduced by comparing the actual rate and the upper bound of
the rate of information transmission for a given channel.
• Shannon has introduced a significant concept of channel capacity
defined as the maximum of mutual information.
• Thus,the channel capacity C is given by
C=Max I(X;Y)=Max[H(X)-H(X/Y)] ...(1)
 ANALOG AND DIGITAL COMMUNICATION

• I(X;Y) is the difference of two entropies and C is max I (X;Y). Hence,

sometimes the unit of I(X,Y) and C is taken as bits/sec.
• The transmission efficiency or channel efficiency is defined as

actual transinformation
η= (or )
max imum transinformation
I ( X ;Y ) I ( X ;Y )
η= = ... ( 2)
max I ( X ;Y ) C

• The redundancy of the channel is defined as

R =1− η
C − I ( X ;Y )
= ... ( 3 )
C
4.9 MAXIMUM ENTROPY FOR CONTINOUS CHANNEL OR GAUSSIAN
CHANNEL
• Probability density function of Gaussian function is given as’
X2
1
P (x ) = e 2σ
2

σ 2π
Where σ2 = Average power of the so urce
The maximum entropy is computed as follows
∞
1
h (x ) = ∫ p ( x ) log dx
P (x )
2
−∞
∞
= − ∫ P ( x ) log 2 P ( x ) dx
−∞

 1 x  2
∞
= − ∫ P ( x ) log 2  e 2σ dx
2

−∞  σ 2π 
 x 
2
∞
= − ∫ P ( x ) log 2 σ 2πe 2σ  dx [  log 2 ( AB ) = log 2 A + log 2 B ]
2

−∞  
 − 2 
2
∞ x

( )
= ∫ P ( x ) log 2 σ 2π + log 2 e 2σ dx
 
−∞

∞ ∞ x2
1
( )
− 2
= ∫ P ( x ) 2 log 2 σ 2π dx + ∫ P ( x ) log 2 e 2σ dx
−∞
2 −∞
SOURCE CODING AND ERROR CONTROL CODING

x2
−log ex 2
[ log 2 e 2 σ2
]=
2σ2
[ n log m=logmn ]
∞
1 log e 2
( )
2
= ∫ P ( x ) log 2 σ 2π dx + ∫ x P ( x ) dx
2 −∞ 2σ2
∞
1 log e
= log 2 ( 2πσ2 ) ∫ P ( x ) dx + 2 ∫
x 2P ( x ) dx
2 −∞
2σ
 ∞ 
 ∫ P ( x ) dx = 1, from properties of pdf 
 −∞ 
∞ 
 ∫ x 2P ( x ) dx = σ2 , from definition of variance 
 −∞ 
1 log e 2
log 2 ( 2πσ2 ) + σ
2 2σ2
1
log 2 ( 2πσ2 ) + log e 
2 
1
h ( x ) = log 2 ( 2πσ2e ) 
2

4.10. CHANNEL CODING THEOREM OR SHANNON’S THEOREM

• The information is transmitted through the channel with rate ‘R’

called information rate.
• Shannon’s theorem says that it is possible to transmit
information with an arbitrarily small probability of error provided
that the information rate ‘R’ is less than or equal to a rate ’C’, called
channel capacity.
• Thus the channel capacity is the maximum information rate with
which the error probability is within the tolerable limits.
Statement
• There exists a coding technique such that the output of the source
may be transmitted over the channel with a probability of error in the
received message which may be made arbitrarily small.
Explanation
• This theorem says that if R≤C,it is possible to transmit information
without any error even if the noise is present.
 ANALOG AND DIGITAL COMMUNICATION

• Coding techniques are used to detect and correct the errors.

Negative statement of channel coding theorem
• Given a source of ‘M’ equally likely messages,with M>>1,which is
generating information at a rate ‘R’,then if’
R>C,
the probability of error is close to unity for every possible set of M
transmitter signals.
• Hence, the negative statement of Shannon’s theorem says that if
R>C,then every message will be in error.

4.10.1 Channel Capacity Theorem (or) Shannon’s Hartley Theorem

(cr)Information Capacity Theorem

• When Shannon’s theorem of channel capacity is applied specifically

to a channel in which the noise is Gaussian is know as Shannon-
Hartley theorem.
• It is also called information capacity theorem.
Statement of theorem
• The channel capacity of a white band limited Gaussian channel is,

C = Blog 2 (1 + S / N ) bite/sec

Where
B →is the channel bandwidth,
S →is the signal power
N →is the total noise power within the channel bandwidth

• We know that the signal power is given as,

B
Power ( P ) = ∫ Power spectral density
−B

No
Here B is bandwidth and power spectral density of white noise is
2
hence noise power N becomes,
B
No
N = ∫
−B
2
df
Noise power
N = NoB
SOURCE CODING AND ERROR CONTROL CODING

T
X Y

Source Destination
N

Figure 4.1 Noisy Communication Channel

Consider a source x and receiver y. As x and y are dependent.

H [ x , y ] = H [y ] + H [ x / y ] ... ( 2)
The noise is added to the system is Gaussian in nature
As source is independent of noise
H [ x , y ] = H [ x ] + H [N ] ... ( 3 )

As y depends on x and N so
Y = f ( x , N ) and Y=x+N
Therefore, H [ x , y ] = H (x , N ) ... ( 4 )
Combining equation(2),(3) and (4)
H [y ] + H [ x / y ] = H [ x ] + H [N ]
H [ x ] − H [ x / y ] = H [y ] − H [N ] ... ( 5 )

We Know that property of mutual information,i.e.,

H [x ] − H [x / y ] = I [x;y ]
Hence , equation (5)becomes,
I [ x ; y ] = H [y ] − H [N ]
Channel Capacity ⇒ C=max I [ x ; y ]
= max H ( y )  − max H ( N )] ... ( 6 )
As noise is gaussian,
max H ( N ) = H ( N ) = log 2 2πe σ2N ... ( 7 )
Where σ2N = N = noisepower
max H ( y ) = log 2 2πe σ2y
Where σ2y = power at receiver
=S +N
= Signal power +Noise power
 ANALOG AND DIGITAL COMMUNICATION

Substitute equation (7),(8)in(6)

C = log 2 2πe σ2y − log 2 2πe σ2N

= log 2 2πe (S + N ) − log 2 2πeN

1/2
 2πe (S + N ) 
= log 2  
 2πeN 
1  S
= log 2 1 + 
2  N

If the signal is band limited, it is sampled at Nyquist rate is given as 2B.

1  S
C = 2B × log 2 1 + 
2  N
 S
C = B log 2 1 +  bits /sec.
 N

Where B is the channel band width. We Know the power spectral density
of noise is
N = NoB
 S 
C = B log 2 1 +  bits /sec
 NoB 

• This is channel capacity of band limited white gaussian noise.

4.10.2 Tradeoff Between Bandwidth and Signal to Noise Ratio
• Channel capacity of the Gaussian channel is given as,
 S
C = B log 2 1 + 
 N 
Above equation shows that the channel capacity depends on two factors.
i. Band width(B) of the channel
ii. Signal to Noise ratio(S/N)
Noiseless channel has infinite capacity
If there is no noise in the channel,then N=0. Hence S/N=∞.Such
a channel is called noiseless channel. Then capacity of such a channel
will be

C = B log 2 (1 + ∞ ) = ∞
SOURCE CODING AND ERROR CONTROL CODING

Thus the noiseless channel has infinite capacity.

Infinite band width channel has limited capacity

• Now if the band width ‘B’ is infinite, the channel capacity is
limited. This is because, as band width increases,noise power (N)
also increases. Noise power is given as
N=NoB

• Due to this increase in noise power, signal to noise (S/N) ratio

decreases. Hence even if B approaches infinity,capacity does not
approach infinity,capacity infinity,As B→∞,capacity approaches an
upper limit. This upper limit is given as,
lim C S
C ∞ = B → ∞1.44
No
Problem 1
The data is to be transmitted at the rate of 10000 bits/sec over a
channel having band width B=3000 Hz .Determine the signal to noise
ratio required. If the band width is increased to 10000 Hz ,then determine
the signal to noise ratio.
Solution
The data is to be transmitted at the rate of 10,000 bits /sec.Hence
channel capacity must be atleast 10000 bits/sec for error-free
transmission.
Channel capacity(C) =10000 bits/sec
Band width(B) =3000Hz
The channel capacity of Gaussian channel,
 S
C = B log 2 1 + 
 N
Putting the values,
 S
10000=3000log 2 1 + 
 N
S
∴ =9
N

Now if the band width is B=10000Hz,then,

 ANALOG AND DIGITAL COMMUNICATION

 S
10000 = 10000 log 2 1 + 
 N
S
∴ =1
N
S
Here , B=3000 =9
N
S
B=10000 =1
N

• Above results show that band width is increases to 10,000 Hz the

signal to noise ratio is reduced by nine times.
• This means the required signal power is reduced,When band width
is increased.

Problem 2
Channel capacity is given by

 S 
C = B log 2 1 +  bits /sec. ... ( 6 )
 N 

In the above equation when the signal power is fixed and white
gaussian noise present,the channel capacity approaches an upper limit
with increase band width ’B’.prove that this upper limit is given as,

lim C S 1 S
C ∞ = B → ∞ = 1.44 =
N o In 2 N o

State Shannon’s information capacity theorem and derive the

expression fo limiting capacity of the channel.
Solution
We Know that,noise power is given as,
N=NoB
Putting this value in equation (1)we get,

 S 
C = B log 2 1 + 
 NoB 
SOURCE CODING AND ERROR CONTROL CODING

By rearranging the above equation we get,

S NoB  S 
C= . log 2 1 + 
No S  NoB 
NoB
S  S  S
log 2 1 + 
No  NoB 
1
S  S  NSB
log 2 1 +  o
No  NoB 

Let us apply the limits as B→∞,

1
lim C S
lim  S  NSB
C∞ = B → ∞ = B → ∞ log 2 1 +  o
No  NoB 
S
In the above equation put x= .Then asB → ∞,x → 0,i.e.,
NoB
S im 1
C∞ = x = 0 log 2 (1 + x )
No x
lim 1
Here let us use the standard relation, x = 0 log 2 (1 + x ) = e,
x
then above equation becomes,
S S log10 e
C∞ = log 2 e =
No N o log10 e
S
1.44
No

This is the required equation. It gives the upper limit on channel

capacity as band width B approaches infinity.
 ANALOG AND DIGITAL COMMUNICATION

Problem 3
A black and white TV picture consists of about 2 x106 Picture
elements with 16 different brightness levels,with equal probabilities. If
pictures are repeated at the rate of 32 per second,calculate average rate
of information conveyed by this TV picture source. If SNR is 30 dB,What
is the maximum band width required to support the transmission of the
resultant video signal

Solution
Given
Picture elements =2x106
Source levels(symbols) =16 i.e.,M=16
Picture repetition rate =32/sec.
S 
  = 30
 N dB
(i) The source symbol entropy(H)
Source emits any one of the 16 brightness levels. Here M=16. These
levels are equiprobable. Hence entropy of such source is given by,
H=log2M
=log216
=4 bits/symbol(level)
(ii)Symbol rate(r)
Each picture consists of 2 x 106 picture elements. Such 32
pictures are transmitted per second. Hence number of picture elements
per second will be,
r = 2 × 106 × 32 symbols/sec
= 64 × 06 symbols/sec

(iii) Average information rate (R):

Information rate of the source is given by

R = rH = 64 × 106 × 4bits /sec.

=2.56 × 108bits /sec.
This is the average rate of by TV picture.

(iv) Required band width for S

= 30dB
N
SOURCE CODING AND ERROR CONTROL CODING

S  S
We Know that   = 10 log10
 N dB N
S
∴ 30=10log10
N
S
∴ =1000
N
Channel coding theorem states that information can be received
without error if,
R≤C
 S
R = 2.56 × 108 and C=Blog 2 1 + 
 N
 S
2.56 × 108 ≤ B log 2 1 + 
 N
i .e., 2.56 × 10 ≤ Blog 2 (1 + 1000 )
8

2.56 × 108
(or ) B≥ i .e., 25.68MH Z
log 2 (1001)
Therefore ,the transmission channel must have a band width of
25.68 MHZ to transmit the resultant video signal.

Problem 4
A voice grade telephone channel has a band width of 3400 Hz.
If the signal to noise ratio(SNR) on the channel is 30 dB,determine the
capacity of the channel. If the above channel is to be used to transmit
4.8 kbps of data determine the minimum SNR required on the channel.
Solution:
Given data: Channel band width B=3400Hz
S 
  = 30dB
 N dB
We Know that
S  S
  = 10 log10
 N dB N
S
∴ 30=10log10
N
S
log10 =3
N
S
∴ =10000
N
 ANALOG AND DIGITAL COMMUNICATION

(i) To calculate capacity of the channel

Capacity of the channel is given as,
 S
C = B log 2 1 + 
 N
=3400log 2 (1 + 1000 )
=33.888 kbits/sec.

(ii) To obtain minimum  S  for 4.8 kbps data

 
N 
Hence the data rate is 4.8 kbps.From channel coding theorem,
R≤C

 S
Here R=4.8 kbps and C=Blog 2 1 + 
 N
Hence above equation becomes,

 S
4.8 kbps ≤ Blog 2 1 + 
 N
 S
i .e., 4800 ≤ 3400log 2 1 + 
 N 
 S
i .e., log 2 1 +  ≥ 1.41176
 N
 S
log10 1 + 
 N  ≥ 1.41176
log10 2
S
∴ ≥ 1.66
N
S 
This means   = 1.66 to transmit data at the rate of 4.8kbps
 N min

Problem 5
For an AWGN channel with 4.0 kHz band width, the noise
spectral density h/2 is 1.0 pico watts/Hz and the signal power at the
receiver is 0.1 mW. Determine the maximum capacity, as also the
actual capacity for the above AWGN channel.
Solution :
Given: B =4000 Hz, S=0.1 x 10-3 W