Superposition and Orthogonality

M. Victor WICKERHAUSER
Washington University in St. Louis, Missouri
[email protected]
http://www.math.wustl.edu/~victor

Mu Alpha Theta National Convention

July 10–15, 2016
A Problem and Its Solution

Problem. Given any arbitrary function f (x), find an accurate

and easy-to-compute formula that approximates it.

Taylor’s Theorem (c.1712) If f (x) is smooth and x is confined

to a bounded interval, then for any desired accuracy there is a
polynomial

p(x) = a0 + a1x + a2x2 + · · · + anxn

that approximates f to that accuracy on the interval.

Note that this is a superposition of the simple functions 1, x, x2, . . .

with weights a0, a1, a2, . . ..

1
A Problem Requiring a More General Solution

The weights are computed from derivatives of the function f ,

for which we use calculus. But what if

• the function is not differentiable?

• the derivatives exists but are expensive to compute?

• the function is known only approximately?

Idea. Use approximate values of the weights that can be com-

puted without differentiation.

2
Two Great Mathematicians, Pure and Applied

Adrien-Marie Legendre (1752–1833) and

Jean-Baptiste Joseph Fourier (1768–1830)
Watercolor by Julien-Leopold Boilly, c.1820.
3
Adrien-Marie Legendre’s Polynomials

P0(x) = 1
P1(x) = x

1
P2(x) = (3x2 − 1)
2
1
P3(x) = (5x3 − 3x)
2
1
P4(x) = (35x4 − 30x2 + 3)
8
1
P5(x) = (63x5 − 70x3 + 15x)
8
...

Relation:
2n + 1 n
Pn+1(x) = xPn(x) − Pn−1
n+1 n+1
4
Graph of the First Six Legendre Polynomials

Legendre polynomials P0 through P5 plotted on their domain

−1 ≤ x ≤ 1.
5
Legendre’s Construction

Theorem 1 Any polynomial p = p(x) may be written as a sum

of Legendre polynomials, multiplied by weights {cn : n = 0, 1, . . .}
specific to p:

p(x) = c0P0(x) + c1P1(x) + c2P2(x) + c3P3(x) + · · ·

Examples:
1 2
x2 = P0(x) + P2(x)
3 3
3 2
x3 = P1(x) + P3(x)
5 5
1 4 8
x4 = P0(x) + P2(x) + P4(x)
5 7 35
3 4 8
x5 = P1(x) + P3(x) + P5(x)
7 9 63
...

6
Application of Legendre’s Construction

Taylor’s polynomial for function f (x) can be written as

p(x) = b0P0(x) + b1P1(x) + · · · + bnPn(x),

where the weights are given by integrals, rather than derivatives:
Z 1
1

bk = k + f (x)Pk (x) dx, k = 0, 1, . . . , n.
2 −1

(This may look just as hard, but in fact integrals are easy to
approximate accurately from just a few values of f (x).)

7
Graph of the First Forty Legendre Polynomials

Legendre polynomials P0 through P39 plotted on their domain

−1 ≤ x ≤ 1.

(Notice that the number of zero-crossings increases with the

degree of the polynomial. Thus degree has some resemblance
to the frequency in sine and cosine functions.)
8
Fourier’s Construction

Theorem 2 Any function f = f (t) may be written as a sum of

sines and cosines, multiplied by numbers {an, bn} specific to f :

f (t) = a0 + a1 cos(t) + a2 cos(2t) + a3 cos(3t) + · · ·

+b1 sin(t) + b2 sin(2t) + b3 sin(2t) + · · ·

Fourier’s weights are also given by integrals:

1 π
Z
a0 = f (x) dx
2π −π
1 π
Z
ak = f (x) cos(kx) dx, k = 1, 2, . . .
π −π
1 π
Z
bk = f (x) sin(kx) dx, k = 1, 2, . . .
π −π

9
Key Ideas

• Simple building blocks: fixed polynomials, or sines and cosines.

• Simple data encoding: one number for each building block.

• Complete and efficient: each function has a unique encoding.

The ingenious choice of orthogonal building blocks, like Legen-

dre’s polynomials or Fourier’s sines and cosines, makes it possible
to compute the weights by integration.

10
Application to Sound and Image Compression

• Audio recordings and images are functions.

• Functions are made of simple building blocks.

• Our senses are imperfect, so approximations suffice.

• Approximations are cheaper.

11
Example of Fourier’s Construction (Good)

Adding up just sines with bn ∼ 1/n2 to get a sawtooth.

Compression: just three terms b1, b3, b5 give the green curve.

12
Example of Fourier’s Construction (Not So Good)

Adding up just sines with bn ∼ 1/n to get a square wave.

Gibbs’ phenomenon: overshoots never go away.

13
Problems with Fourier’s Construction

• Infinitely many numbers {an, bn} are needed to represent a

given function f , and some simple functions require very
many for a good approximation.

• Sines and cosines are not localized, so that any error in a

weight appears as error everywhere.

• Even if the function f is continuous, its Fourier series may

not converge.

14
Two More Great Mathematicians

Alfréd Haar (1885–1933) and Ingrid Daubechies (1954– )

15
Time and Frequency Content Analyzed Together

∆x

F ∆ξ
r
(x,ξ)
e
q
u
Information
e cells
n
c
y

Time

Signal

16
Waveforms Localized in Time and Frequency

17
History B.D. [Before Daubechies]

• Fourier bases (1822, Paris)

• Haar bases (1910, Math. Annalen)

• Gabor functions (1946, J. IEE)

• Balian-Low theorem (1981, CRAS)

• Wilson bases (1987, Cornell)

18
Ingrid Daubechies’ Construction

Theorem 3 Any function f = f (t) may be written as a sum of

def
wavelets wjk (t) = w(2j t+k), multiplied by numbers cjk specific
to f :
X X
f (t) = cjk wjk (t),
j∈Z k∈Z
and the mother wavelet w = w(t) can be chosen with these three
properties:

Smoothness: w and its first d derivatives w′, w′′, . . . , w(d) are

continuous functions.

Compact support: w(t) is zero at all |t| > 5d.

Orthogonality: The set {wjk : j, k ∈ Z} is an orthonormal basis.

19
Some Nice Wavelets
1

0.8

0.6

0.4

0.2

-0.2

-0.4
0 20 40 60 80 100 120

Six dilations and translations, on an interval, of a particular

mother wavelet (9,7-biorthogonal symmetric).
20
History A.D. [After Daubechies]

• Lapped orthogonal transforms (1990, IEEE ASSP)

• Biorthogonal wavelets, wavelet packets (1992, IEEE IT)

• WSQ fingerprint standard (1993, FBI)

• Wavelets on spheres (1995, ACM)

• The lifting implementation (1996, ACHA; 1998, JFAA; )

• JPEG-2000 compression (1999)

21
Example Images

22
Close Up of Correlated Pixels

23
Two-Dimensional Waveforms I

24
Two-Dimensional Waveforms II

25
Two-Dimensional Waveforms III: JPEG vs. JPEG-2000

26
Transform Coding Image Compression

Compression:

Scanned Transform Quantize Code Storage

image

Decompression:

Storage Decode Unquantize Untransform Restored

image

27
Parts Description

Compression:

Transform: convert pixels to amplitudes;

Quantize: round off the amplitudes to small numbers;
Code: remove redundancy from the small number sequence.

Decompression:

Decode: expand to recover the small number sequence;

Unquantize: insert an amplitude for each small number;
Untransform: recover pixels from approximate amplitudes.

28
Wavelet Transform: Multiresolution Signal Splitting

hx g
hh gh
hhh ghh h g

ghhh

hhhh

Split signal x into averages hx and details gx.

Replace x ← hx and repeat

29
Multiresolution Image Splitting

Picture (at top) becomes thumbnail (at bottom left) plus two
layers of saved details (highlighted).

30
Storage of Multiresolution Image Data

31
Custom Compression Algorithms

Sample 1 S ∑
image u
1 m

Sample 2 o
image f Best-basis search
2
s
q
.. .. .. u
. . . a
Sample N r
image e
N s

Training algorithm for a custom transform coding image

compression algorithm.

32
Good Bases for Images I

Five-level wavelet basis, used in JPEG-2000.

33
Good Bases for Images II

Five-level wavelet packet basis, used in WSQ.

34
Compression Sometimes Improves Things

Rough Radiation Dose Approximation in 2D:

4 M particle simulation

35
...By Eliminating the Rough Errors

Improved Approximation in 2D:

Compressed 4 M particle simulation

36
...If the Right Amount of Compression is Done
x 0.001
11

10.5

10

9.5

RMS error
9

8.5

7.5

7
0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.11 0.12
Threshold ( ε )

Deasy et al., Fig. 3

Reduction in RMS error by a rough approximation compressed

toward a smooth target, by wavelet threshold.
37
...Which, Fortunately, is Easy to Find.
0.14

0.12

0.1

Best threshold ( ε)
0.08

0.06

0.04

0.02

0
2 4 8 16
Source electrons (millions)

Deasy, et al., Fig. 4

Best wavelet thresholds for compression from a rough

approximation.
38
Example: Rough Radiation Dose Approximation – 1D

4 M particle simulation — 1D cross-section, close up.

39
Example: Compressed Approximation –1D

Compressed 4 M particle simulation — 1D cross-section, close

up.
40
Some Notable Works

• Ingrid Daubechies. “Orthonormal Bases of Compactly

Supported Wavelets.” Comm. Pure Appl. Math.
41(1988),909–996.

• Ingrid Daubechies. Ten Lectures on Wavelets. CBMS-NSF

Regional Conference Series in Applied Mathematics. SIAM
Press, Philadelphia, 1992.

• Albert Cohen, Ingrid Daubechies and Jean-Christophe

Feauveau. “Biorthogonal Bases of Compactly Supported
Wavelets” Comm. Pure Appl. Math. 45(1992),485–500.

• Ingrid Daubechies and Wim Sweldens. “Factoring Wavelet

Transforms into Lifting Steps.” Fourier Anal. Appl.
4:3(1998),245–267.

41