Approaching Shannon
Ruediger Urbanke, EPFL
Summer School @ USC, August 6th, 2010
Many thanks to Dan Costello, Shrinivas Kudekar, Alon Orlitsky, and Thomas Riedel
for their help with these slides.
Storing Shannon
Networking Shannon
Completing Shannon
Compressing Shannon
Reading Shannon
Coding
Disclaimer
Technical slides do not contain references.
These are all summarized at the end of each section.
Classes of Codes
(linear) block codes
sparse graph codes
convolutional codes
polar codes
How Do We Compare?
capacity
rate
block length
PN (R, C)
block error probability
complexity
How We Compare: Error Exponent
1
log(PN (R, C))
N N
E(R, C) = lim
error exponent
How We Compare: Finite-Length Scaling
f (z) scaling function; mother curve
10-2
PN (R, )
10-3
10-4
10
1/
0
N in
cre
asin
-5
10-6
10-7
10
threshold
10-1
-8
N 1/ ( )
scaling exponent
channel quality
How We Compare: Finite-Length Scaling
f (z) scaling function; mother curve
PN (R, C)
10-2
10-3
10-4
10
1/
0
N in
cre
asin
-5
10-6
10-7
threshold
10-1
10-8
N 1/ (C
C C)
C
scaling exponent
rate
Finite-Length Scaling
lim
N :N 1/ (CR)=z
PN (R, C) = f (z)
f (z)
scaling function; mother curve
>0
scaling exponent
Finite-Length Scaling -- References
V. Privman, Finite-size scaling theory, in Finite Size Scaling and Numerical Simulation of Statistical
Systems, V. Privman, ed., World Scientific Publ., Singapore, 1990, pp. 198.
Complexity
exponential versus polynomial
=C R
gap to capacity
linear -- but look at prefactor
Block Codes
Error Exponent of Block Codes under MAP
Figure ``borrowed from
1
log(PN (R, C)) error exponent
N N
E(R, C) = lim
quadratic
Error Exponent -- References
R. Gallager, Information Theory and Reliable Communication, Wiley 1968.
A. Barg and G. D. Forney, Jr., Random codes: Minimum distances and error exponents, IEEE
Transactions on Information Theory, Sept 2002.
Scaling of Block Codes under MAP -- BEC
``perfect code
PN
0
=0
distribution of erasures
PN
=1R
erasure fraction
=1
E[E] = N
2 ] = N (1 )
E[(E E)
E N (N , N (1 ))
N ((1 R) )
N ((1 R) )
Q
=Q
=Q
N (1 )
(1 )
(1 )
z = N ((1 R) )
Scaling of Block Codes under MAP -- BEC
random linear block codes are almost perfect
00101010001
01110010010
10101010101
01000101001
01011100010
probability that full rank
n1
i=0
square binary
random matrix
of dimension n
n1
2n 2i
in n
=
(1
2
) 0.28878809508 . . .
2n
i=0
00101010001
01110010010
10101010101
01000101001
if we have k rows
less then probability
decays by roughly
(k+1
2 )
hence for random linear block codes the transition is of constant (on an absolute scale) width
Scaling of Block Codes under MAP
log A(N, P ) = N C
N V Q1 (P ) + O(log(N ))
block length
error probability
A(N, P )
size of largest such code
C = E[i(x, y)]
i(x, y) = log
V = V[i(x, y)]
dp(y | x)
dp(y)
Finite-Length Scaling -- References
G. Landsburg, Uber eine Anzahlbestimmung und eine damit zusammenhangende Reihe, J.
Reine Angew. Math. vol. 111, pp. 87-88, 1893.
A. Feinstein, A new basic theorem of information theory, IRE Trans. Inform. Theory, vol.
PGIT-4, pp. 222, 1954.
V. Strassen, Asymptotische Abschtzungen in Shannons Informationstheorie, Trans. Third
Prague Conf. Information Theory, pp. 689723, 1962.
Y. Polyanskiy, H. V. Poor and S. Verd, "Dispersion of Gaussian Channels," 2009 IEEE Int.
Symposium on Information Theory, Seoul, Korea, June 28-July 3, 2009.
Y. Polyanskiy, H. V. Poor and S. Verd, "Dispersion of the Gilbert-Elliott Channel," 2009 IEEE
Int. Symposium on Information Theory, Seoul, Korea, June 28-July 3, 2009.
For a very simple proof of previous result ask Thomas Riedel, UIUC
convolutional codes
Convolutional Codes
Convolutional Codes
Figures ``borrowed from
affine
Finite-Length Scaling of LDPC Codes -- BEC
scaling behavior?
K constraint length
Convolutional Codes -- Some References
Big bang:
P. Elias, Coding for noisy channels, in IRE International Convention Record, Mar. 1955, pp. 3746.
Algorithms and error exponents:
J. M. Wozencraft, Sequential decoding for reliable communication, Research Lab. of Electron. Tech. Rept. 325,
MIT, Cambridge, MA, USA, 1957.
R. M. Fano, A heuristic discussion of probabilistic decoding, IEEE Trans. Information Theory, vol. IT-9, pp.
64-74, Apr. 1963.
A. J. Viterbi, Error bounds of convolutional codes and an asymptotically optimum decoding algorithm, IEEE
Trans. Inform. Theory, 13 (1967), pp. 260269.
H. L. Yudkin, Channel state testing in information decoding, Sc.D. thesis, Dept. of Elec. Engg.,M.I.T., 1964.
J. K. Omura, On the Viterbi decoding algorithm, IEEE Trans. Inform. Theory,15 (1969), pp. 177179.
G. D. Forney, Jr., The Viterbi algorithm, Proc. IEEE, 61 (1973), pp. 268278.
K. S. Zigangirov, Time-invariant convolutional codes: Reliability function, in Proc. 2nd Joint Soviet-Swedish
Workshop Information Theory, Grnna, Sweden, Apr. 1985.
N. Shulman and M. Feder, Improved Error Exponent for Time-Invariant and Periodically Time-Variant
Convolutional Codes, IEEE Trans. Inform. Theory, 46 (2000), pp. 97103.
G. D. Forney, Jr., The Viterbi algorithm: A personal history. E-print: cond-mat/0104079, 2005.
Overview:
A. J. Viterbi and J. K. Omura, Principles of Digital Communication and Coding, McGraw-Hill, New York, NY,
USA, 1979.
S. Lin and D. J. Costello, Jr., Error Control Coding, Prent. Hall, Englewood Cliffs, NJ, USA, 2nd ed., 2004.
R. Johannesson and K. S. Zigangirov, Fundamentals of Convolutional Coding, IEEE Press, Piscataway, NJ, USA,
1999.
Some Open Questions
Scaling behavior
digrams such as TH, ED, etc. In the second-order approximation, digram structure is introduced. After a
letter is chosen, the next one is chosen in accordance with the frequencies with which the various letters
follow the first one. This requires a table of digram frequencies pi j . In the third-order approximation,
trigram structure is introduced. Each letter is chosen with probabilities which depend on the preceding two
letters.
3. T HE S ERIES OF A PPROXIMATIONS TO E NGLISH
To give a visual idea of how this series of processes approaches a language, typical sequences in the approximations to English have been constructed and are given below. In all cases we have assumed a 27-symbol
alphabet, the 26 letters and a space.
1. Zero-order approximation (symbols independent and equiprobable).
XFOML RXKHRJFFJUJ
QPAAMKBZAACIBZLUDREVOIGRES
LDPC ZLPWCFWKCYJ FFJEYVKCQSGHYD
HJQD.
2. First-order approximation (symbols independent but with frequencies of English text).
OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA
NAH BRL.
3. Second-order approximation (digram structure as in English).
ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANDY TOBE SEACE CTISBE.
4. Third-order approximation (trigram structure as in English).
IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTURES OF THE REPTAGIN IS REGOACTIONA OF CRE.
5. First-order word approximation. Rather than continue with tetragram, , n-gram structure it is easier
and better to jump at this point to word units. Here words are chosen independently but with their
appropriate frequencies.
REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES
THE LINE MESSAGE HAD BE THESE.
6. Second-order word approximation. The word transition probabilities are correct but no further structure is included.
THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT
THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.
The resemblance to ordinary English text increases quite noticeably at each of the above steps. Note that
these samples have reasonably good structure out to about twice the range that is taken into account in their
construction. Thus in (3) the statistical process insures reasonable text for two-letter sequences, but fourletter sequences from the sample can usually be fitted into good sentences. In (6) sequences of four or more
words can easily be placed in sentences without unusual or strained constructions. The particular sequence
of ten words attack on an English writer that the character of this is not at all unreasonable. It appears then
that a sufficiently complex stochastic process will give a satisfactory representation of a discrete source.
The first two samples were constructed by the use of a book of random numbers in conjunction with
(for example 2) a table of letter frequencies. This method might have been continued for (3), (4) and (5),
since digram, trigram and word frequency tables are available, but a simpler equivalent method was used.
Sparse Graph Codes
LDPC Ensemble
n
o
d
e
s
x1+x4+x8 = 0
p
e
r
m
u
t
a
t
i
o
n
c
h
e
c
k
n
o
d
e
s
Hx=0
sparse
v
a
r
i
a
b
l
e
(1-(1-x)5)3
Asymptotic Analysis -- BEC
34
(1-(1-x)5)3
MAP versus BP
35
Capacity Achieving -- BEC
Capacity Approaching -- BMS
Error Exponent of LDPC Codes -- BP
P {|PN (G, , ) E[PN (G, , )]| } eN
graph
channel parameter
# of iterations
If E[PN (G, , )] converges to zero for large and if code
has error correcting radius then we can prove that the code
has an error exponent under iterative decoding.
simplest sufficient condition: code has expansion at least 3/4 which is
true whp if left degree is at least 5; (less restrictive conditions are known
but more complicated); codes used in ``practice do not have error exponents
Expansion
(dl, dr)-regular cannot
have expansion beyond
(dl-1)/dl
|C|
at most size
dl |V|
|V|
take the smallest ratio
|C|/(dl |V|)
over all ``small sets
remarkably, random graphs
essentially achieve this bound whp
Finite-Length Scaling of LDPC Codes -- BEC
PN = Q(z/)(1 + O(N
13
))
z=
scaling parameters computable
(x) =
1
5
x + x3
6
6
N (BP N 2/3 )
(we ignore error floor here!)
(x) = x5
R = 3/7
PN
10-1
10-2
= 0.5791
/ = 0.6887
10-3
10-4
10-5
0.3
0.35
0.4
0.45
0.5
BP = 0.4828
Optimization
10-1
10 11 12 13
10
40.58 %
0.0
10-2
rate/capacity
1.0
contribution to error floor
10-3
10 12 14 16 18 20 22 24 26
10-4
10-5
10-6
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
Finite-Length Scaling of LDPC Codes -- BAWGNC
same form of scaling law; parameters are computable but no proof
(3, 6) BSC
(3, 4) BAWGNC
Gap To Threshold versus Length
lim
N :N 1/ (CR)=z
PN (R, C) = f (z)
halving the gap requires increasing length by 4
1/
N1/
N
(C R) = z
N = (z/)
=2
fixes error
additive gap
Gap versus Complexity (per bit)
BEC/Threshold -- O(1); degrees are constant and we touch every edge at most
once
BEC/Capacity -- O(log(1/\delta)) for standard LDPC; degrees grow like log(1/\delta)
and we touch every edge once
BEC/Capacity -- O(1) for MN-type LDPC ensembles; degrees are constant and we
touch every edge at most `once
BMS/Threshold -- ???
BMS/Capacity -- ???
Sparse Graph Codes -- Some References
Big bang:
R. G. Gallager, Low-density parity-check codes, IRE Trans.Inform.Theory, 8 (1962).
C. Berrou, A. Glavieux, and P. Thitimajshima, Near Shannon limit error-correcting coding and
decoding, in Proc. of ICC, Geneva, Switzerland, May 1993.
Analysis:
M. Luby, M. Mitzenmacher, A. Shokrollahi, and D. A. Spielman, Analysis of low density codes and
improved designs using irregular graphs, in Proc. of the 30th Annual ACM Symposium on Theory of
Computing, 1998, pp. 249258.
M. Luby, M. Mitzenmacher, A. Shokrollahi, and D. A. Spielman, Efficient erasure correcting codes, IEEE
Trans. Inform. Theory, 47 (2001), pp. 569584.
T. Richardson, A. Shokrollahi, and R. Urbanke, Design of capacity-approaching irregular low-density
parity-check codes, IEEE Trans. Inform. Theory, 47 (2001), pp. 619637.
T. Richardson and R. Urbanke, The capacity of low-density parity check codes under messagepassing decoding, IEEE Trans. Inform. Theory, 47 (2001), pp. 599618.
S.-Y. Chung, G. D. Forney, Jr., T. Richardson, and R. Urbanke, On the design of low-density paritycheck codes within 0.0045 dB of the Shannon limit, IEEE Commun. Lett., 5 (2001), pp. 5860.
Error exponents:
D. Burshtein and G. Miller, Expander graph arguments for message-passing algorithms, IEEE Trans.
Inform. Theory, 47 (2001), pp. 782790.
O. Barak and D. Burshtein, Upper Bounds on the Error Exponents of LDPC Code Ensembles, The
IEEE International Symposium on Information Theory (ISIT-2006), Seattle, July 2006.
Sparse Graph Codes -- Some References
Finite-length scaling:
A. Montanari, Finite-size scaling of good codes, in Proc. of the Allerton Conf. on Commun., Control,
and Computing, Monticello, IL, USA, Oct. 2001.
A. Amraoui, A. Montanari, T. Richardson, and R. Urbanke, Finite-length scaling for iteratively decoded
LDPC ensembles, in Proc. of the Allerton Conf. on Commun., Control, and Computing, Monticello, IL,
USA, Oct. 2003.
J. Ezri, A. Montanari, and R. Urbanke, Finite-length scaling for Gallager A, in 44th Allerton Conf. on
Communication, Control, and Computing, Monticello, IL, Oct. 2006.
A. Dembo and A. Montanari, Finite size scaling for the core of large random hyper-graphs. E-print:
math.PR/0702007, 2007.
Ezri, J., Montanari, A., Oh, S. and Urbanke, R. (2008) The Slope Scaling Parameter for General
Channels, Decoders, and Ensembles. Proceeding of the IEEE International Symposium on Information
Theory.
Complexity:
A. Khandekar and R. J. McEliece, On the complexity of reliable communication on the erasure
channel, in Proc. IEEE Int. Symp. Information Theory (ISIT2001), Washington, DC, Jun. 2001, p. 1.
H. D. Pfister, I. Sason, and R. Urbanke, Capacity-achieving ensembles for the binary erasure channel
with bounded complexity, IEEE Transactions on Inform. Theory, vol. 51 , issue 7, 2005 , pp. 2352 2379.
Overviews:
D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge Univ. Press,
2003.
T. Richardson and R. Urbanke, Modern Coding Theory, Cambridge Univ. Press, 2008.
Some Open Questions
Simple design procedures?
Can you achieve capacity on general BMS channels?
Thresholds under LP decoding?
Scaling for general BMS channels?
Scaling under MAP?
Scaling under LP decoding?
Scaling under flipping decoding?
Scaling to capacity?
Polar Codes
patterns
Codes from Kronecker Product of G2
Reed-Muller Codes
choose rows of largest weight
Definition of Channels
Polar Codes
W -- BMS channel
Definition of Channels
Channel Polarization
0
0
bad channels
Definition of Channels
Successive Decoding
Successive Decoding
Stefan Meier http://ipgdemos.epfl.ch/polarcodes/
threshold
Channel Polarization
Stefan Meier http://ipgdemos.epfl.ch/polarcodes/
How Do Channels Polarize?
X1
known
noise
X2
BEC()
BEC()
U1 =X1+X2; observe Y1 and Y2
U2 =X2; U2 =X1+U1 ; observe Y1 and Y2
U1
U2
+
X1
1-(1-)2
much worse
X2
Y1 Y2
2 much better
X2
parity-check node
X1+U1=X1
Y2 Y1
repetition code
total capacity = (1-)2+1-2=2(1-)
Definition of Channels
How Do Channels Polarize?
0.9375 0.9961 0.75
0.5
0.8789
0.43750.8086 0.25
0.3164
0.5625 0.6836
0.1914
0.1211
0.0625
0.0039
0.5
Polar Codes -- Some References
Big bang:
E. Arikan, Channel polarization: A method for constructing capacity-achieving codes for symmetric
binary-input memoryless channels, http://arxiv.org/pdf/0807.3917
Exponent:
E. Arikan and E. Telatar, On the Rate of Channel Polarization, http://arxiv.org/pdf/0807.3806
S. B. Korada, E. Sasoglu, and R. Urbanke, Polar Codes: Characterization of Exponent, Bounds, and
Constructions, http://arxiv.org/pdf/0901.0536
Source Coding:
N. Hussami, S. B. Korada, and R. Urbanke, Performance of Polar Codes for Channel and Source
Coding, http://arxiv.org/pdf/0901.2370
S. B. Korada, and R. Urbanke, Polar Codes are Optimal for Lossy Source Coding, http://arxiv.org/pdf/
0903.0307
E. Arikan, Source Polarization, http://arxiv.org/pdf/1001.3087
Non-symmetric and non-binary channels:
E. Sasoglu, E. Telatar, and E.Arikan, Polarization for arbitrary discrete memoryless channels, http://
arxiv.org/pdf/0908.0302
R. Mori and T. Tanaka, Channel Polarization on q-ary Discrete Memoryless Channels by Arbitrary
Kernels, http://arxiv.org/pdf/1001.2662
R. Mori and T. Tanaka, Non-Binary Polar Codes using Reed-Solomon Codes and Algebraic Geometry
Codes, http://arxiv.org/pdf/1007.3661
MAC channel:
E. Sasoglu, E. Telatar and Edmund Yeh, Polar codes for the two-user multiple-access channel, http://
arxiv.org/pdf/1006.4255
E. Abbe and E. Telatar, Polar Codes for the m-User MAC and Matroids, http://arxiv.org/pdf/
1002.0777
Compound channel:
S. H. Hassani, S. B. Korada, and R. Urbanke, The Compound Capacity of Polar Codes, http://
arxiv.org/pdf/0907.3291
Wire-tap channel and security:
H. Mahdavifar and A. Vardy, Achieving the Secrecy Capacity of Wiretap Channels Using Polar Codes,
http://arxiv.org/pdf/1007.3568
E. Hof and S. Shamai, Secrecy-Achieving Polar-Coding for Binary-Input Memoryless Symmetric WireTap Channels, http://arxiv.org/pdf/1005.2759
Mattias Andersson, Vishwambhar Rathi, Ragnar Thobaben, Joerg Kliewer, Mikael Skoglund, Nested
Polar Codes for Wiretap and Relay Channels, http://arxiv.org/pdf/1006.3573
O. O. Koyluoglu and H. El Gamal, Polar Coding for Secure Transmission and Key Agreement, http://
arxiv.org/pdf/1003.1422
Constructions:
R. Mori and T. Tanaka, Performance and Construction of Polar Codes on Symmetric Binary-Input
Memoryless Channels, http://arxiv.org/pdf/0901.2207
M. Bakshi, S. Jaggi, and M. Effros, Concatenated Polar Codes, http://arxiv.org/pdf/1001.2545
Scaling:
S. H. Hassani and R. Urbanke, On the scaling of Polar codes: I. The behavior of polarized channels,
http://arxiv.org/pdf/1001.2766
T. Tanaka and R. Mori, Refined rate of channel polarization, http://arxiv.org/pdf/1001.2067
S. H. Hassani, K. Alishahi and R. Urbanke, On the scaling of Polar Codes: II. The behavior of unpolarized channels, http://arxiv.org/pdf/1002.3187
Error Exponent of Polar Codes -- BEC
A First Guess
Z ,
1 (1 Z)2 ,
wp
wp
1
2,
1
2.
Z 2,
2Z Z 2 ,
wp 12 ,
wp 12 .
Y = log(Z)
2Y,
Y 1,
wp 12 ,
wp 12 .
X = log(Y )
X + 1,
X,
wp 12 ,
wp 12 .
assume that Z is already small,
hence Y is large
Error Exponent of Polar Codes -- BEC
A First Guess
X + 1,
X,
wp 12 ,
wp 12 .
random walk on lattice with drift
after m steps we expect X to have value roughly m/2
this means we expect Y to have value roughly 2
m/2
this means we expect Z to have value roughly 2
=2
Error Exponent of Polar Codes
lim P (Zm 2
2m/2+
mQ1 (R/C)/2+o(
m)
)=R
Finite-Length Scaling for Polar Codes (BEC)
wp 12 ,
wp 21 .
Z 2,
1 (1 Z)2 ,
Battacharyya process
z=0
z=1
1
2
= 1/2
symmetry of distribution
(general case follows in similar manner)
QN (x) =
z=0
1
1
(i)
|{i : x E(WN ) }|
N
4
z=1
z = 2x
1
2
Finite-Length Scaling for Polar Codes -- BEC
2Q2N (x) = QN (1/2 1/4 x/2)+(121{x1/8} )QN (min x/2, 1/2 x/2)
1
Q(x) = lim N QN (x)
N
Q(x) = 2
1
1
Q(1/2
scaling assumption
1/4 x/2) + (1 21{x1/8} )Q(min x/2, 1/2 x/2)
solve this functional equation (numerically)
this gives Q(x) (up to scaling) and
Q(x)
BEC
3.62
BAWGNC
Finite-Length Scaling for Polar Codes -- BEC
Simulations versus Scaling
log10 PN (R, C)
PN (R, C) Q1 (N (C R))
N 1/ (C R)
C R
N = 223 , 224 , 225 , 226
Gap To Capacity versus Length
0.43 bits/channel use
86 % of capacity
lim
N :N 1/ (CR)=z
PN (R, C) = f (z)
1/
N1/
N
(C R) = z
N = (z/)
fixes error
additive gap
! 10 billion to get 1% close to capacity
Gap to Capacity versus Complexity
= O(log(1/))
complexity per bit
Some Open Questions
Variation on the theme that performs better at small lengths?
Do RM codes achieve capacity?
Make scaling conjecture under successive decoding rigorous.
Scaling behavior under MAP decoding?
Find a reasonable channel where they do not work. :-)
Message
sparse graph codes -- best codes in ``practice; still miss
some theory; error floor region is tricky; still somewhat of an
art to construct
polar codes -- nice for theory; not (yet) ready for applications
but the field is young; how do we improve finite-length
performance
scaling behavior is the next best thing to exact analysis;
probably more meaningful characterization for practical
case than error exponent
