Information Theory

Communication System
Information
Transmitter Channel Receiver Destination
Source
Processed Noisy
Message Estimate of Message
Signal Signal

• Goal of Communication System is to transmit information from

source to destination.
• Information Source
▪ generates random messages for the transmission .
▪ Types of Messages are: Audio signal , Video signal, Image and Text
▪ Types of Information Source
▪ The Memory less Source:
– Generates Independent random messages .
▪ The Markov Source (Source with Memory):
– Generate dependent random messages
– Each symbol depends on previous symbol or state of source
 Communication System
Information
Transmitter Channel Receiver Destination
Source
Processed Noisy
Message Estimate of Message
Signal Signal

➢ Channel
▪ act as a medium of transmission.
▪ Channel may be wired or wireless.
❖ Wired communication System
▪ Telephone communication system (channel: Two wire transmission line)
▪ Cable TV transmission system (Coaxial cable as a channel)
▪ Local area network (LAN) (Ethernet cables as a channel)
❖ Wireless communication system
▪ Free space as a medium of transmission (channel)
▪ Cellular network (Mobile communication system)
▪ Satellite communication system
▪ FM, AM and TV broadcast system
 Communication System
Information
Transmitter Channel Receiver Destination
Source
Processed Noisy
Message Estimate of Message
Signal Signal

➢ The major objective of any communication system is to transmit

information with
▪ Maximum data rate
▪ Minimum probability of symbol error
▪ Minimum transmitted power

➢ Now the questions are:

– how much maximum data rate can be supported by any
communication system for reliable communication?
– What is the measure of information?
 Information Theory
• In particular, Information theory answers two fundamental
questions:
– What is the irreducible complexity, below which a signal cannot be
compressed?
• (Entropy of a source)
– What is the maximum transmission rate for reliable communication
over a noisy channel?
• (Capacity of a channel)

• In general, Information theory deals

– with the mathematical modeling and analysis of a communication
system.
– with the amount of information content in any random signal
(Information Measure)
– Relationship between
• Signal to Noise Ratio and Channel Capacity
• Channel bandwidth and Capacity.
 INFORMATION AND UNCERTAINTY
Consider following sentences:

• The Sun rise in the east and set in the west

– ( Highly probable event , Probability is 1 )

• Dog bites a man

– (It happen but not frequently-- associated with some probability or
uncertainty )

• Man bites a dog

– (Rarest event -- Highly uncertain – less probability)

• The Thar desert will receive heavy rainfall in this rainy season
– (Rarest event -- Highly uncertain– less probability)
 INFORMATION AND UNCERTAINTY
• Intuitive Feel:
– Information contained in an event is closely related to its uncertainty
or probability

– Messages with low probability (i.e., highly unexpected messages)

contains a large amount of information.

– Messages with high probability deliver very low amount of

information.

– Before the event occurs,

• there is some amount of uncertainty.

– After the occurrence of the event,

• there is no uncertainty anymore, but there is gain in the amount of
information.

The amount of information should be inversely

proportional to the probability of the message.
 Amount of Information
(Mathematical Definition)

• Let 𝑥𝑗 is an event with a probability 𝑝 𝑥𝑗 , then the amount of

information associated with 𝑥𝑗 is given as –
1
𝐼 𝑥𝑗 = 𝑙𝑜𝑔𝑏
𝑝 𝑥𝑗
Self Information

• The choice of base for the logarithm define different units of

information.
– When the base is ‘2’, then the unit of information is – bit

– When the base is ‘e’ then the unit of information is – nat

– When the base is ‘10’, then the unit of information is – decit or Hartley

𝟏 𝒉𝒂𝒓𝒕𝒍𝒆𝒚 = 𝟑. 𝟑𝟐 𝒃𝒊𝒕𝒔, and 𝟏 𝒏𝒂𝒕 = 𝟏. 𝟒𝟒 𝒃𝒊𝒕𝒔

 Amount of Information
(Mathematical Definition)

Self Information 1
𝐼 𝑥𝑗 = 𝑙𝑜𝑔𝑏
𝑝 𝑥𝑗
This definition satisfied following properties

1. 𝑰 𝒙𝒋 = 𝟎, 𝒇𝒐𝒓 𝒑 𝒙𝒋 = 𝟏
– Certain events convey no information

2. 𝑰 𝒙𝒋 ≥ 𝟎, 𝒇𝒐𝒓 𝟎 ≤ 𝒑 𝒙𝒋 ≤ 𝟏
– Every event provides some information

3. 𝑰 𝒙𝒋 > 𝑰 𝒙𝒌 , 𝒇𝒐𝒓 𝒑 𝒙𝒋 < 𝒑(𝒙𝒌 )

– Less probable event convey more information

4. 𝑰 𝒙𝒋 , 𝒚𝒌 = 𝑰 𝒙𝒋 + 𝑰 𝒚𝒌 , 𝒇𝒐𝒓 𝒊𝒏𝒅𝒆𝒑𝒆𝒏𝒅𝒆𝒏𝒕 𝒆𝒗𝒆𝒏𝒕

 Why Logarithm
• The logarithmic measure is justified by the desire for information to be
additive.

• When two independent events take place then the total information is
sum of information in each event. (Fourth Property)

𝐼 𝑥𝑗 , 𝑦𝑘 = 𝐼 𝑥𝑘 + 𝐼 𝑦𝑘 , 𝑓𝑜𝑟 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑒𝑣𝑒𝑛𝑡

• But the probabilities of independent events multiply to give their

combined probabilities,

• Therefore, we must take Logarithms in order for the joint probability of

independent events Or messages to contribute additively to the
information gained.
 Why Logarithm
• Let 𝑥𝑗 and 𝑦𝑘 are two independent event with a probability 𝒑 𝒙𝒋 and 𝒑 𝒚𝒌 respectively and self

information associated with them is 𝐼 𝑥𝑗 and 𝐼 𝑦𝑘

𝟏 𝟏
• Let 𝑰 𝒙𝒋 = 𝒇 and 𝑰 𝒚𝒌 = 𝒇 ----- some function of probability
𝒑 𝒙𝒋 𝒑 𝒚𝒌

• Then the probability of the joint event is define as –

𝒑 𝒙𝒋 , 𝒚𝒌 = 𝒑 𝒙𝒋 𝒑 𝒚𝒌

• Information associated with this joint event is define as

𝟏 𝟏
𝑰 𝒙𝒋 , 𝒚𝒌 = 𝒇 =𝒇
𝒑 𝒙𝒋 , 𝒚𝒌 𝒑 𝒙𝒋 𝒑 𝒚𝒌

• This total Information 𝐼 𝑥𝑗 , 𝑦𝑘 must be equal to the sum of individual information 𝐼 𝑥𝑗 and 𝐼 𝑦𝑘
𝟏 𝟏 𝟏
𝑰 𝒙𝒋 , 𝒚𝒌 = 𝒇 =𝒇 +𝒇
𝒑 𝒙𝒋 𝒑 𝒚𝒌 𝒑 𝒙𝒋 𝒑 𝒚𝒌

• From above expression , it is clearly visible that we require a function which converts multiplication into
addition and Logarithm is one such function.

𝟏 𝟏 𝟏
𝑰 𝒙𝒋 , 𝒚𝒌 = 𝒍𝒐𝒈 = 𝒍𝒐𝒈 + 𝒍𝒐𝒈 = 𝑰 𝒙 𝒋 + 𝑰 𝒚𝒌
𝒑 𝒙𝒋 𝒑 𝒚𝒌 𝒑 𝒙𝒋 𝒑 𝒚𝒌
 Entropy
• Consider a information source which generates M- discrete messages.
• Let us define the output of this source as a discrete random variable
𝑿 which may take values from 𝒙𝟏 𝒕𝒐 𝒙𝑴 .
• These discrete messages 𝑥1 𝑡𝑜 𝑥𝑀 may be called as alphabet of source.
• The Entropy of the source X is defined as the statistically averaged
amount of information ---mean of I(X)
1
𝐸𝑛𝑡𝑟𝑜𝑝𝑦 𝑜𝑓 𝑆𝑜𝑢𝑟𝑐𝑒 = 𝐻 𝑋 = 𝐸 𝐼 𝑋 = 𝐸 log
𝑝 𝑥𝑗
𝑀 𝑀
1
𝐻 𝑋 = ෍ 𝑝(𝑥𝑗 ) 𝐼 𝑥𝑗 = ෍ 𝑝 𝑥𝑗 log 𝑏
𝑝 𝑥𝑗
𝑗=1 𝑗=1

𝑴
𝟏
𝑯 𝑿 = ෍ 𝒑 𝒙𝒋 𝒍𝒐𝒈𝟐 𝒃𝒊𝒕𝒔/𝒔𝒚𝒎𝒃𝒐𝒍
𝒑 𝒙𝒋
𝒋=𝟏
 Properties of Entropy
• 𝑯 𝑿 ≥𝟎

(Since 𝑰 𝒙𝒋 ≥ 𝟎 , Every event provides some information and never bring a loss of information)

• Lower Bound: 𝑯 𝑿 = 𝟎
– if and only if the probability of one symbol in a set is 1 and remaining
symbols in the set have zero probability.

– This lower bound corresponds to No Uncertainty

• Upper Bound: 𝑯 𝑿 = 𝒍𝒐𝒈𝒃 𝑴

– If and only if the probabilities of all the symbols in the set are equal, 𝒑 𝒙𝒋 =
𝟏
, 𝒇𝒐𝒓 𝒂𝒍𝒍 𝑴
𝑴

– This upper bound corresponds to Maximum Uncertainty

• 𝑯 𝑿 𝒊𝒔 bounded as -- 𝟎 ≤ 𝑯 𝑿 ≤ 𝒍𝒐𝒈𝒃 𝑴
 Example-1
• Consider a binary source with the probabilities
𝑝 𝑥1 = 𝑝 and 𝑝 𝑥2 = 1 − 𝑝
• The Entropy of this source is given as –
𝑀
1
𝐻 𝑋 = ෍ 𝑝 𝑥𝑗 𝑙𝑜𝑔2
𝑝 𝑥𝑗
𝑗=1

1 1
𝐻 𝑋 = 𝑝 log 2 + 1 − 𝑝 log 2
𝑝 1−𝑝

• At p = 0.5 ,
1 1
𝐻 𝑋 = 0.5 log 2 + 1 − 0.5 log 2 =1
0.5 0.5
• Similarly it can be shown that for a source with M-symbols,
the Entropy is maximum when all the symbols have equal
probability.

• The maximum Entropy is given as

𝑴
𝟏
𝑯 𝑿 = ෍ 𝒑 𝒙𝒋 𝒍𝒐𝒈𝟐
𝒑 𝒙𝒋
𝒋=𝟏

𝑴
𝟏 𝟏
𝑯𝒎𝒂𝒙 𝑿 = ෍ 𝒍𝒐𝒈𝟐 𝑴 = 𝒍𝒐𝒈𝟐 𝑴 × 𝑴
𝑴 𝑴
𝒋=𝟏

𝑯𝒎𝒂𝒙 𝑿 = 𝒍𝒐𝒈𝟐 𝑴
 Example-2
• Consider a source which generates 4 discrete symbol with the
probabilities
𝑝 𝑥1 = 1/2 , 𝑝 𝑥2 =1/4,
𝑝 𝑥3 = 1/8 and 𝑝 𝑥4 =1/8

• The Entropy of this source is given as -

1 1 1 1
𝐻 𝑋 = log 2 2 + log 2 (4) + log 2 8 + log 2 (8)
2 4 8 8

= 𝟏. 𝟕𝟓 𝒃𝒊𝒕𝒔 𝒑𝒆𝒓 𝒔𝒚𝒎𝒃𝒐𝒍

• If the probability of all symbols are equal, then Entropy of this source is
given as
1 1 1 1
𝐻 𝑋 = log 2 4 + log 2 4 + log 2 4 + log 2 4
4 4 4 4
1
= 4 ∗ log 2 4 = 𝒍𝒐𝒈𝟐 𝟒 = 𝟐 𝒃𝒊𝒕𝒔 𝒑𝒆𝒓 𝒔𝒚𝒎𝒃𝒐𝒍 = 𝒍𝒐𝒈𝟐 𝑴
4
 Entropy
• The Entropy of the source X is defined as the statistically averaged
amount of information ---mean of I(X)
1
𝐸𝑛𝑡𝑟𝑜𝑝𝑦 𝑜𝑓 𝑆𝑜𝑢𝑟𝑐𝑒 = 𝐻 𝑋 = 𝐸 𝐼 𝑋 = 𝐸 log
𝑝 𝑥𝑗

𝑴
𝟏
𝑯 𝑿 = ෍ 𝒑 𝒙𝒋 𝒍𝒐𝒈𝒃 𝒖𝒏𝒊𝒕𝒔/𝒔𝒚𝒎𝒃𝒐𝒍
𝒑 𝒙𝒋
𝒋=𝟏

𝑴
𝟏
𝑯 𝑿 = ෍ 𝒑 𝒙𝒋 𝒍𝒐𝒈𝟐 𝒃𝒊𝒕𝒔/𝒔𝒚𝒎𝒃𝒐𝒍
𝒑 𝒙𝒋
𝒋=𝟏

• 𝑯 𝑿 𝒊𝒔 bounded as -- 𝟎 ≤ 𝑯 𝑿 ≤ 𝒍𝒐𝒈𝒃 𝑴
𝟎 ≤ 𝑯 𝑿 ≤ 𝒍𝒐𝒈𝟐 𝑴
 Derivatives of Entropy
• Entropy defined for one set of random variable is suitable either for
transmitter or receiver.

• However, to study the behavior of communication system, we required to

study both transmitter and receiver simultaneously.

• For this, we require a pair of random variable which represent both

transmitter and receiver side symbols.

• For such pair of random variables, Now we will define following:

– Joint Probabilities,
– Conditional Probabilities,
– Joint Entropy
– Conditional Entropy
– Mutual Information
 Joint Probability
• Consider a pair of random variables (X and Y)

𝑋 = 𝑥1 , 𝑥2 , … , 𝑥𝑀 and 𝑌 = 𝑦1 , 𝑦2 , … , 𝑦𝑁
Symbols at the transmitter Symbols at the receiver

𝑥1 𝑦1 ⋯ 𝑥1 𝑦𝑁
• 𝑋𝑌 -- represent joint event 𝑋𝑌 = ⋮ ⋱ ⋮
𝑥𝑀 𝑦1 ⋯ 𝑥𝑀 𝑦𝑁

• Probability associated with X – 𝑷 𝑿 = 𝒑 𝒙𝒋 = 𝒑 𝒙𝟏 , 𝒑 𝒙𝟐 , … , 𝒑(𝒙𝑴 )

• Probability associated with Y – 𝑷 𝒀 = 𝒑 𝒚𝒌 = 𝒑 𝒚𝟏 , 𝒑 𝒚𝟐 , … , 𝒑(𝒚𝑵 )

• Probability associated with joint event XY -

𝒑 𝒙𝟏 , 𝒚𝟏 ⋯ 𝒑 𝒙𝟏 𝒚𝑵
𝑷 𝑿𝒀 = 𝒑 𝒙𝒋 , 𝒚𝒌 = ⋮ ⋱ ⋮
𝒑 𝒙𝑴 𝒚𝟏 ⋯ 𝒑 𝒙𝑴 𝒚𝑵
 Joint Entropy
• The Joint Entropy 𝐻 𝑋, 𝑌 is defined as:
1
𝐻 𝑋, 𝑌 = 𝐸 𝑙𝑜𝑔
𝑃 𝑋, 𝑌

𝑀 𝑁
1
𝐻 𝑋, 𝑌 = ෍ ෍ 𝑝 𝑥𝑗 , 𝑦𝑘 log
𝑝 𝑥𝑗 , 𝑦𝑘
𝑗=1 𝑘=1
 Conditional Entropy
• The Conditional Entropy 𝑯 𝑿/𝒀 represents the amount of uncertainty
remaining about the random variable X after the random variable Y has been
observed.
• It is defined as -
1
𝐻 𝑋/𝑌 = 𝐸𝑋,𝑌 𝑙𝑜𝑔
𝑃 𝑋/𝑌
𝑴 𝑵
𝟏
𝑯 𝑿/𝒀 = ෍ ෍ 𝒑 𝒙𝒋 , 𝒚𝒌 𝒍𝒐𝒈 𝒙𝒋
𝒋=𝟏 𝒌=𝟏 𝒑
𝒚𝒌

• Similarly , the Conditional Entropy 𝑯 𝒀/𝑿 can also be written as –

𝑴 𝑵
𝟏
𝑯 𝒀/𝑿 = ෍ ෍ 𝒑 𝒙𝒋 , 𝒚𝒌 𝒍𝒐𝒈
𝒚𝒌
𝒋=𝟏 𝒌=𝟏 𝒑
𝒙𝒋
 Relationship Between Different Entropies
❖ . Conditioning reduces entropy
𝐻 𝑋/𝑌 ≤ 𝐻 𝑋

❖ . If X and Y are independent, then

𝐻 𝑋/𝑌 = 𝐻 𝑋

❖. 𝐻 𝑋, 𝑌 = 𝐻 𝑋 + 𝐻 𝑌/𝑋

❖. 𝐻 𝑋, 𝑌 = 𝐻 𝑌 + 𝐻 𝑋/𝑌

𝑋 𝑌
❖. 𝐻 𝑋 −𝐻 𝑌 =𝐻 − 𝐻
𝑌 𝑋
 𝑀

𝐻 𝑋 = − ෍ 𝑝 𝑥𝑗 log 2 𝑝 𝑥𝑗 Entropy of the transmitter

𝑗=1

𝑴 𝑵
𝟏
𝑯 𝑿, 𝒀 = ෍ ෍ 𝒑 𝒙𝒋 , 𝒚𝒌 𝒍𝒐𝒈 Joint Entropy
𝒑 𝒙𝒋 , 𝒚𝒌
𝒋=𝟏 𝒌=𝟏

𝑴 𝑵
𝟏
𝑯 𝑿/𝒀 = ෍ ෍ 𝒑 𝒙𝒋 , 𝒚𝒌 𝒍𝒐𝒈 𝒙𝒋 Conditional Entropy
𝒋=𝟏 𝒌=𝟏 𝒑
𝒚𝒌

𝑴 𝑵
𝟏 Conditional Entropy
𝑯 𝒀/𝑿 = ෍ ෍ 𝒑 𝒙𝒋 , 𝒚𝒌 𝒍𝒐𝒈
𝒚𝒌
𝒋=𝟏 𝒌=𝟏 𝒑
𝒙𝒋

𝒀 𝑿 Chain Rule
𝐇 𝑿, 𝒀 = 𝑯 𝑿 + 𝑯 =𝑯 𝒀 +𝑯
𝑿 𝒀
23
 Mutual Information
• Goal of communication system is to transfer information from transmitter
to receiver.
• Transmitted symbol 𝑥𝑗 contain some information which is associated with
the probability 𝑝(𝑥𝑗 ), which is also known as a priori probability of 𝒙𝒋 .
• Thus before the reception of symbol 𝑦𝑘 , the information or uncertainty
about 𝑥𝑗 is given by
𝐼 𝑥𝑗 = − log 𝑝(𝑥𝑗 )
• At the receiver, after receiving the symbol 𝑦𝑘 , the information related to
𝑥𝑗 is now associated with the conditional probability 𝑝 𝑥𝑗 /𝑦𝑘 which is
also known as – a posteriori probability.
• Thus , the information or uncertainty about 𝑥𝑗 after receiving symbol 𝑦𝑘 is
given as –

− log 𝑝 𝑥𝑗 /𝑦𝑘
24
 Mutual Information
• The information gained about symbol 𝑥𝑗 after receiving the symbol 𝑦𝑘 is the
net reduction in its uncertainty.
• This reduction in uncertainty of 𝑥𝑗 is known as Mutual Information.
• Mathematically it is defined as
Mutual Information = Initial uncertainty – final uncertainty

𝐼 𝑥𝑗 , 𝑦𝑘 = − log 𝑝 𝑥𝑗 − − log 𝑝 𝑥𝑗 /𝑦𝑘

𝒙𝒋
𝒑
𝒚𝒌
𝑰 𝒙𝒋 , 𝒚𝒌 = 𝐥𝐨𝐠
𝒑 𝒙𝒋

Using Bayes rule

𝑥𝑗 𝒑 𝒙 𝒋 , 𝒚𝒌
𝑝 𝑥𝑗 , 𝑦𝑘 =𝑝 𝑝 𝑦𝑘 𝑰 𝒙𝒋 , 𝒚𝒌 = 𝒍𝒐𝒈
𝑦𝑘 𝒑 𝒙𝒋 𝒑(𝒚𝒌 )

25
 Average Mutual Information
• The amount of mutual information exchange between X and Y is the average
of 𝐼 𝑥𝑗 , 𝑦𝑘 .
𝑀 𝑁

𝐼 𝑋, 𝑌 = 𝐸 𝐼 𝑥𝑗 , 𝑦𝑘 = ෍ ෍ 𝑝 𝑥𝑗 , 𝑦𝑘 𝐼 𝑥𝑗 , 𝑦𝑘
𝑗=1 𝑘=1

𝑀 𝑁
𝑝 𝑥𝑗 , 𝑦𝑘
𝐼 𝑋, 𝑌 = ෍ ෍ 𝑝 𝑥𝑗 , 𝑦𝑘 𝑙𝑜𝑔
𝑝 𝑥𝑗 𝑝(𝑦𝑘 )
𝑗=1 𝑘=1

26
 Properties of Mutual Information
• For statistically independent symbols, there is no mutual information, i.e,
𝑰 𝒙𝒋 , 𝒚𝒌 = 𝟎

𝑝 𝑥𝑗 , 𝑦𝑘
𝐼 𝑥𝑗 , 𝑦𝑘 = 𝑙𝑜𝑔
𝑝 𝑥𝑗 𝑝(𝑦𝑘 )

For independent symbols 𝒑 𝒙𝒋 , 𝒚𝒌 = 𝒑 𝒙𝒋 𝒑 𝒚𝒌

𝑝 𝑥𝑗 𝑝 𝑦𝑘
𝐼 𝑥𝑗 , 𝑦𝑘 = 𝑙𝑜𝑔 = 𝑙𝑜𝑔 1 = 0
𝑝 𝑥𝑗 𝑝(𝑦𝑘 )
• If 𝒙𝒋 = 𝒚𝒌 then the mutual information is equal to the self information

𝑝 𝑦𝑘 , 𝑦𝑘 𝑝 𝑦𝑘 1
𝐼 𝑦𝑘 , 𝑦𝑘 = 𝑙𝑜𝑔 = 𝑙𝑜𝑔 = 𝑙𝑜𝑔
𝑝 𝑦𝑘 𝑝(𝑦𝑘 ) 𝑝 𝑦𝑘 𝑝(𝑦𝑘 ) 𝑝 𝑦𝑘

𝐼 𝑦𝑘 , 𝑦𝑘 = 𝐼 𝑦𝑘
27
 Properties of Mutual Information
• Non-Negativity : It is always positive, that is , 𝐼 𝑋, 𝑌 ≥ 0

• Symmetry: It is symmetric, that is, 𝐼 𝑋, 𝑌 = 𝐼 𝑌, 𝑋

• Expansion of Mutual Information: It is the reduction in the uncertainty of

X due to the knowledge of Y.

𝑰 𝑿, 𝒀 = 𝑯 𝑿 − 𝑯 𝑿/𝒀 = 𝑯 𝒀 − 𝑯 𝒀/𝑿

• For the same sources, the mutual information becomes Entropy

𝐼 𝑋, 𝑋 = 𝐻 𝑋
• Relative Entropy: It is also known as relative entropy between joint
distribution and the product of marginal distributions
𝑀 𝑁
𝑝 𝑥𝑗 , 𝑦𝑘
𝐼 𝑋, 𝑌 = ෍ ෍ 𝑝 𝑥𝑗 , 𝑦𝑘 𝑙𝑜𝑔
𝑝 𝑥𝑗 𝑝(𝑦𝑘 )
𝑗=1 𝑘=1

28
 Relationship Between Mutual Information and Joint Entropy

Using Bayes rule

𝑥𝑗
𝑝 𝑥𝑗 , 𝑦𝑘 = 𝑝 𝑝 𝑦𝑘
𝑦𝑘

29
Relationship Between Mutual Information and Joint Entropy

30
 Summary
𝒀
𝐇 𝑿, 𝒀 = 𝑯 𝑿 + 𝑯
𝑿

𝑿
𝐇 𝑿, 𝒀 = 𝑯 𝒀 + 𝑯
𝒀

𝑿
𝐈 𝑿, 𝒀 = 𝑯 𝑿 − 𝑯
𝒀

𝒀
𝐈 𝑿, 𝒀 = 𝑯 𝒀 − 𝑯
𝑿
31
 Reference Book

• S. Haykin, Digital Communication Systems, John Wiley & Sons, NY,

2014.

• R. Bose, Information theory, coding and cryptography, Publisher: Tata

McGraw-Hill, 3rd Edition 2016.

• R P Singh and S D Sapre, Communication System (Analog and Digital),

Tata McGraw-Hill, 3rd Edition 2012.
Thank you