Input

Source Encoder

Channel Encoder

Binary Interface Output

Channel

Source Decoder

Channel Decoder

Separation of source and channel coding.

REASONS FOR BINARY INTERFACE

Standardization (Simplies implementation)

Layering (Simplies conceptualization) Loses nothing in performance (Shannon says)

input sampler waveform

quantizer

discrete coder

analog sequence output waveform analog lter table lookup

symbol sequence discrete decoder

reliable binary Channel

Layering of source coding

input sampler waveform

quantizer

discrete coder

analog sequence output waveform analog lter table lookup

symbol sequence discrete decoder

reliable binary Channel

A waveform source is usually sampled or

expanded into a series, producing a sequence
of real or complex numbers.

input sampler waveform

quantizer

discrete coder

analog sequence output waveform analog lter table lookup

symbol sequence discrete decoder

reliable binary Channel

The analog sequence is encoded by

quantization into sequence of symbols.

input sampler waveform

quantizer

discrete coder

analog sequence output waveform analog lter table lookup

symbol sequence discrete decoder

reliable binary Channel

Both analog and discrete sources then require

binary encoding of sequence of symbols.

DISCRETE SOURCE CODING OBJECTIVE: Map sequence of symbols into binary sequence with unique decodability. SIMPLEST APPROACH: Map each source sym bol into an L-tuple of binary digits. Choose L as smallest integer satisfying 2L M , i.e., log2 M L < log2 M + 1; L = dlog2 M e

yellow, purple, magenta}): red 000 blue 001 green 010 yellow 011 purple 100 magenta 101 This can be easily decoded. Example:

Example (for alphabet {red, blue, green,

the ASCII code maps let

ters, numbers, etc. into bytes. These are called xed length codes.

FIXED-TO-FIXED LENGTH SOURCE

CODES
Segment source symbols into n-tuples. Map each n-tuple into binary L-tuple where log2 M n L < log2 M n + 1; L = dn log2 M e

Let L = L be number of bits per source symbol n log2 M L < log2 M + 1 n

VARIABLE LENGTH SOURCE CODES Motivation: Probable symbols should have shorter codewords than improbable to reduce bpss. A variable-length source code C encodes each
symbol x in source alphabet X to a binary
codeword C(x) of length l(x). For example, for X = {a, b, c} C(a) = 0
C(b) = 10
C(c) = 11

10

Decoder must parse the received sequence. Requires unique decodability: For every string of source letters {x1, x2, . . . , xn}, the encoded output {C(x1)C(x2), . . . , C(xn)} must be distinct, i.e., must dier from {C(x0 )C(x0 ), . . . , C(x0 )} for m 1 2 any other source string {x0 , . . . , x0 }. m 1 If C(x1) C(xn) = C(x0 ) C(x0 ), decoder must m 1 fail on one of these inputs. We will show that prex-free codes are uniquely decodable.

11

Unique Decodability: For every string of source letters {x1, x2, . . . , xn}, the encoded output {C(x1)C(x2), . . . , C(xn)} must be distinct, i.e., must dier from {C(x0 )C(x0 ), . . . , C(x0 )} for any other m 1 2 source string {x0 , . . . , x0 }. m 1 If C(x1) C(xn) = C(x0 ) C(x0 ), decoder must m 1 fail on one of these inputs. Example: Consider a 0, b 01, c 10 Then ac 010 and ba 010, Not uniquely decodable.

12

Unique Decodability: For every string of source letters {x1, x2, . . . , xn}, the encoded output {C(x1)C(x2) C(xn)} must be distinct, i.e., must dier from {C(x0 )C(x0 ) C(x0 )} for any other m 1 2 source string {x0 , . . . , x0 }. m 1 If C(x1) C(xn) = C(x0 ) C(x0 ), decoder must m 1 fail on one of these inputs. Example: Consider a 0, b 01, c 11 Then accc 0111111=016; bccc 01111111=017. This can be shown to be uniquely decodable.

13

PREFIX-FREE CODES
A code is prex-free if no codeword is a prex of any other codeword. A prex of a string y1, . . . , yk is y1, . . . , yi for any i k. A prex-free code can be represented by a bi nary tree which grows from left to right; leaves represent codewords.
1

b
1 0

1 0

a0 b 11 c 101

Every codeword is at a leaf, but not all leaves are codewords. Empty leaves can be short ened. A full code tree has no empty leaves.

14

 PP PP PP PP PP PP PP PP PP PP PP PP



P
PPP PP PP PP PP

ba

1 0

ca

bb bc cb cc

ab ac

aa

Prex-free codes are uniquely decodable: Construct a tree for a concatenation of codewords. To decode, start at the left, and parse when ever a leaf in the tree is reached.

15

THE KRAFT INEQUALITY

The Kraft inequality is a test on the existence of prex-free codes with a given set of codeword lengths {l(x), x X }. Theorem (Kraft): Every prex-free code for an alphabet X with codeword lengths {l(x), x X } satises
xX

2l(x) 1

(1)

Conversely, if (1), then a prex-free code with lengths {l(x)} exists. Moreover, a prex-free code is full i (1) is satised with equality.

16

We prove this by associating codewords with base 2 expansions i.e., decimals in base 2. Represent binary codeword y1, y2, . . . , ym as .y1y2 ym = y1/2 + y2/4 + + ym2m 1.0

Interval [1/2, 1) 1 .1 01 .01 00 .00

Interval [1/4, 1/2)

Interval [0, 1/4)

17

Represent binary codeword y1, y2, . . . , ym as .y1y2 ym = y1/2 + y2/4 + + ym2m 1.0

Interval [1/2, 1) 1 .1
01 .01
00 .00

Interval [1/4, 1/2)

Interval [0, 1/4)

C(aj ) is a prex of C(ai) if and only if the expan sion of C(aj ) contains the expansion of C(ai) in its approximation interval.

18

1.0

Interval [1/2, 1) 1 .1 01 .01 00 .00

Interval [1/4, 1/2)

Interval [0, 1/4)

C(aj ) is a prex of C(ai) if and only if the expan sion of C(aj ) contains the expansion of C(ai) in its approximation interval. Thus a code is a prex code i the base 2 approximation intervals are disjoint.

19

1.0

Interval [1/2, 1) 1 .1
01 .01
00 .00

Interval [1/4, 1/2)

Interval [0, 1/4)

A code is a prex code i the base 2 approxi mation intervals are disjoint. But the sum of disjoint approximation intervals is at most 1.

20

1.0

Interval [1/2, 1) 1 .1
01 .01
00 .00

Interval [1/4, 1/2)

Interval [0, 1/4)

The sum of disjoint approximation intervals is at most 1. Code is full i approximation intervals ll up
[0, 1)

21

DISCRETE MEMORYLESS SOURCES

The source output is an unending sequence, X1, X2, X3, . . . , of randomly selected letters from a nite set X , called the source al phabet. Each source output X1, X2, . . . is selected from X using a common probability mea sure. Each source output Xk is statistically inde pendent of the other source outputs X1, . . . , Xk1, Xk+1, . . . .

22

Probability Structure for Discrete Sources

English text: e, i, and o are far more probable
than q, x, and z.
Successive letters are dependent; (th and qu).
Some letter strings are words, others are not.
Long term grammatical constraints.
The discrete memoryless source is a toy model that can be easily generalized after understand ing it.

23

PREFIX-FREE CODES FOR DMS

Let l(x) be the length of the codeword for let ter x X . Then L(X) is a random variable (rv) where L(X) = l(x) for X = x. Thus L(X) = l(x) with probability pX (x). E(L) = L =
X
x

pX (x)l(x)

Thus L is the number of encoder output bits per source symbol.

24

OBJECTIVE: choose integers {l(x)} subject to

Kraft to minimize L.
Let X = {1, 2, . . . , M } with pmf p1, . . . , pM .
Denote the unknown lengths by l1, . . . , lM .
Lmin = min P
2li 1
=1 i

l1,... ,lM :

M X

Forget about the lengths being integer for now. P Minimize Lagrangian: i(pili + 2li ). @
P l i(pili + 2 i )

pili

@li

= pi (ln 2)2li = 0

Choose so that the optimizing {li} satisfy P l i 2 i = 1. Then any other choice of {li} sat isfying constraint will have a larger L.

25

@li If we choose = 1/ ln 2, then pi = 2li li = log pi Lmin(nonint.) =

X
i

P l i(pili + 2 i )

= pi (ln 2)2li = 0

pi log pi = H(X)

H(X) is called the entropy of the rv X. We will see that it is the minimum number of bi nary digits per symbol needed to represent the source. For now, it is a lower bound for prex-free codes.

26

Theorem: Entropy bounds Let Lmin be the minimum expected codeword length over all prex-free codes for X. Then H(X) Lmin < H(X) + 1 Lmin = H(X) i each pi is integer power of 2. Proof of H(X) L for prex-free codes: Let l1, . . . , lM be codeword lengths. X 1 X p log i pili H(X) L = i i pi X 2li = p log , i i pi

27

u1
1

ln u

The inequality ln u u1 or log u (log e)(u1).

This inequality is strict except at u = 1.
H(X) L = =
X
i i X h

p log

2li pi
i

2li pi log e 0
i

p i i

"

2li pi

1 log e

Equality occurs i pi = 2li for each i.

28

Theorem: Entropy bound for prex-free codes: H(X) Lmin < H(X) + 1 Lmin = H(X) i each pi is integer power of 2. Proof that Lmin < H(X) + 1: Choose li = d log(pi)e. Then li < log(pi) + 1 li log(pi) so so Lmin L < H(X) + 1
X
i

2li

X
i

pi = 1

29