MMAT5390: Mathematical Image Processing

Chapter 5: Image Enhancement in the Spatial Domain

In Chapter 4, we have talked about how we can enhance an image in the frequency domain. In
this chapter, we will talk about how image enhancement can be done in the spatial domain.

1 Image denoising in the spatial domain

We will first discuss image denoising algorithms in the spatial domain.

1.1 Image denoising by linear filter

Definition 1.1. Linear filter is a process to modify the pixel value by a linear combination of
the pixels values of its local neighbourhood.

Example 1.2. Let f be an N × N image. Extend the image periodically. Modify f to f˜ by:

f˜(x, y) = f (x, y) + 3f (x − 1, y) + 2f (x + 1, y).

This is a linear filter.

Example 1.3. Define

1
f˜(x, y) = (f (x + 1, y) + f (x − 1, y) + f (x, y + 1) + f (x, y − 1))
4
This is also a linear filter.

Linear filter versus discrete convolution

We consider an image f to be defined on [−M, M ] × [−N, N ]. So, f is a (2M + 1) × (2N + 1)
image.
Assume f is also periodically extended. Let H be another (2M + 1) × (2N + 1) matrix defined on
[−M, M ] × [−N, N ].
Recall: The discrete convolution is defined as:
M
X N
X
I ∗ H(u, v) = I(u − m, v − n)H(m, n)
m=−M n=−N

(Linear combination of pixel values around (u, v))

Therefore, Linear filter is equivalent to a discrete convolution.

Remark. 1. DF T (I ∗ H) = C · DF T (I) · DF T (H) for some constant C. Therefore, the linear

filter is equivalent to modifying the DFT coefficients by multiplication.

1
 2. A geometric illustration of the idea is as follows.
ˆ
I  I
 
∗H
 ∼
=  × Ĥ

y y

I˜ −−
←−−−DF
−− −−
− −T−
−− −−
−−
−−
→− Iˆ˜
iDF T
(spatial) (frequency)

Example 1.4. In Example 1.2, if f is defined on [−M, M ] × [−N, N ], then:

f˜ = f ∗ H

where  
0 2 0
H= 0 1 0 
0 3 0

In Example 1.3, f˜ = f ∗ H where  

0 1 0
1
H=  1 0 1 
4
0 1 0

Remark. H is called the filter.

Some commonly used linear filters:

• Mean filter:  
1 1 1
1
H= 1 1 1 
9
1 1 1

(Here, we only write down the entries of the matrix for indices −1 ≤ k, l ≤ 1 for simplicity.
All other matrix entries are equal to 0.)
This is called the mean filtering with window size 3 × 3.
Below is an example of mean filtering on an image with impulse noise:

2
 Below is an example of mean filtering on an image with Gaussian noise:

r2
 
• Gaussian filter: The entries of H are given by the Gaussian function g(r) = exp − 2 ,
p 2σ
where r = x2 + y 2 .
Below is an example of Gaussian filtering on an image:

(Please refer to “Lecture 17 powerpoint” for some more examples of mean filter and Gaussian
filter on real images)

Properties of linear filtering

• Associativity: A ∗ (B ∗ C) = (A ∗ B) ∗ C
• Commutativity: I ∗ H = H ∗ I
• Linearity:
(s · I) ∗ H = I ∗ (s · H) = s · (I ∗ H)
(I1 + I2 ) ∗ H = (I1 ∗ H) + (I2 ∗ H)

3
Remark. 1. Advantage of Gaussian filter: We can check that convolution of Gaussian filter is
again Gaussian (with larger σ).

2. Thus, successive Gaussian filter = Gaussian filter with a larger σ (because of the property of
associativity).
Proof of associativity

N −1
mn
X
[(x ∗ y) ∗ z](n) = ∗ y) ∗ z(m)ej2π
(x\ N

m=0
N −1
mn
X
= ∗ y(m)ẑ(m)ej2π
C xd N

m=0
N −1
mn
X
= C 2 x̂(m)ŷ(m)ẑ(m)ej2π N

m=0
N −1
mn
X
= ∗ z(m)ej2π
C x̂(m)yd N

m=0
N −1
mn
X
= (y ∗ z)(m)ej2π N
x ∗\
m=0
= (x ∗ (y ∗ z))(n)

1.2 Generalization of linear filter: Non-linear (spatial) filter

Median filter

window for me-

dian filter

Take a window with center at pixel (x0 , y0 ). Update the pixel value at (x0 , y0 ) from I(x0 , y0 ) to
˜ 0 , y0 ) = median(I within the window)
I(x

Example 1.5. If the pixel values within a window are 0, 0, 1, 2, 3, 7, 8, 9, 9, then the pixel value is
updated as 3 (median).

Below is an example of median filtering on an image:

4
Below is a comparison of mean and median filtering on an image:

Edge-preserving filter

various windows of
fixed size

• Step 1: Consider all windows with certain size around pixel (x0 , y0 ) (not necessarily centered

5
 at (x0 , y0 ));
• Step 2: Select a window with minimal variance;
• Step 3: Do a linear filter (mean filter, Gaussian filter and so on).

Below is an example of edge preserving filtering on an image (compared with other types of
filtering):

Non-local mean filter

Let g be a N × N image. Let X = (x, y) and X 0 = (x0 , y 0 ) be two different pixels of g. Consider
two corresponding neighbourhoods:

SX = {(x + s, y + t) : −a ≤ s, t ≤ a} ; SX 0 = {(x0 + s, y 0 + t) : −a ≤ s, t ≤ a}

∴ Sizes of SX and SX 0 are both (2a + 1) × (2a + 1).

| {z } | {z }
=m =m

Denote gX = g|SX and gX 0 = g|SX 0 , then gX and gX 0 are two m × m small images.
We call gX and gX 0 the local patch of g at X and X 0 respectively.
Apply Gaussian filter (linear) to gX and gX 0 to get g̃X and g̃X 0
Define the least square distance between two local patches as:

||g̃X − g̃X 0 ||2 = sum of squares of coefficients of the matrix (g̃X − g̃X 0 )

−||g̃X − g̃X 0 ||2

 
0
Define the weight by : w(X, X ) = exp , where h is called the noise level param-
h2
eter.
Definition 1.6. The non-local mean filter of an image g is given by:
0 0
P
X 0 ∈ image domain w(X, X )g(X )
ĝ = P 0
X 0 ∈ image domain w(X, X )

where ĝ is the output image.

Remark:

6
 • The image g is often: 1. Periodically extended; or 2. set outside region to have zero pixel
values.

• The weight is smaller if the overall intensities over a local patch at X and X 0 are different.

Below is an example of non-local mean filtering on an image:

(Please see “Lecture 18 powerpoint” for more illustration of non-local mean filter on real images)

1.3 Image denoising by Anisotropic Diffusion

Consider the following partial differential equation:

 2
∂ I(x, y; σ) ∂ 2 I(x, y; σ)

∂I(x, y; σ)
=σ +
∂σ ∂x2 ∂y 2
= σ∇ · (∇I(x, y; σ0)) (∗)

∂v1 ∂v2
(where ∇· = divergence defined by ∇ · ((v1 , v2 )) = + and
  ∂x ∂y
∂I ∂I
∇I = gradient of I = , )
∂x ∂y
Then, the Gaussian function with standard deviation σ,
 2
x + y2

1
g(x, y; σ) = exp −
2πσ 2 2σ 2

satisfies equation (*).

In fact, the Gaussian linear filter is approximately solving the heat diffusion equation.
Given an image I(x, y) (which is assumed to be continuous image over the whole 2D domain), the
Gaussian filter is equivalent to the convolution of I with the Gaussian function:
e y; σ) = I ∗ g(x, y; σ)
I(x,
Z ∞Z ∞
:= g(x − u, y − v; σ)I(u, v) du dv
−∞ −∞
Z ∞Z ∞
= g(u, v; σ)I(x − u, y − v) du dv
−∞ −∞

7
This is analogous to the discrete convolution:
XX
I ∗ J(u, v) = I(u − m, v − n)J(m, n)
m n

Z ∞ Z ∞
∂ Ie ∂g(u, v; σ)
∴ = I(x − u, y − v) du dv
∂σ −∞ −∞ ∂σ
Z ∞Z ∞
∂ 2 g(u, v; σ)
=σ I(x − u, y − v) du dv
−∞ −∞ ∂u2
∞ Z ∞
∂ 2 g(u, v; σ)
Z
+σ I(x − u, y − v) du dv
−∞ −∞ ∂v 2
Z ∞Z ∞
∂2
=σ 2 g(u, v; σ)I(x − u, y − v) du dv
∂x −∞ −∞
Z ∞Z ∞
∂2
+σ 2 g(u, v; σ)I(x − u, y − v) du dv
∂y −∞ −∞
= σ∇ · (∇I(x,
e y; σ))

Remark. • Here, we us the fact that:

∂ ∂g(x, y; σ)
(g(x, y; σ) ∗ f ) = ∗f
∂x ∂x
• Gaussian filtering ≈ solving the above partial differential equation.
• Refer to powerpoint for the solution of the heat diffusion equation at different scales

Anisotropic diffusion for edge-preserving image denoising

The general form of a heat diffusion equation can be written as:
∂I(x, y; σ)
= ∇ · (K(x, y)∇I(x, y; σ))
∂σ
where K controls the rate of diffusion at (x, y) (smaller K = smaller diffusion at (x, y)).
Motivation: Less diffusion on the edges.
Edge detector: Edges of an image can be determined by |∇I(x, y)|.
If (x, y) is on the edges, |∇I(x, y)| is big while if (x, y) is in the interior of the object, |∇I(x, y)| ≈ 0
(Assume the image is piecewise constant)
Suitable K to preserve edges:

1
1. K(x, y) = (not good as ∇I can be ~0)
|∇I(x, y)|
1
Modification: K(x, y) = for ε small
|∇I(x, y)| + ε2
 
|∇I(x, y; σ)|
2. K(x, y) = exp −
b
Remark. We can choose K such that K depends on |∇I| and K ≈ 0 when |∇I| is big.
Hence, the anisotropic diffusion algorithm for solving the image de-noising problem can be written
as:    
∂I(x, y; σ) |∇I(x, y; σ)|
= ∇ · exp − ∇I(x, y; σ) (∗∗)
∂σ b

In the discrete case, (**) can be considered as:

|∇I n (x, y)|
   
n+1 n n
I (x, y) − I (x, y) = D1 exp − D2 I (x, y)
b

8
where D1 = linear operator approximating ∇· and
D2 = linear operator approximating the gradient ∇

Starting with I 0 (x, y) = I(x, y), which is the original image, we iteratively de-noise the original
image I(x, y) := I 0 (x, y). Such a process is called anisotropic diffusion image denoising.
Below is an example of (edge-preserving) anisotropic diffusion on an image:

1.4 Image denoising via energy minimization

Let g be the noisy image. We will consider additive noise. So, g can be written as:

g(x, y) = f (x, y) + n(x, y)

where f (x, y) is the clean image and n(x, y) is the noise.

Recall: Laplacian marking: g = f − ∆f obtains a sharp (non-smooth) image from a smooth image.
Conversely, to get a smooth image f from a non-smooth image g, we can solve f such that:
f = g + ∆f (Partial differential equation)
The equation can be discretized to get:

f (x, y) = g(x, y) + [f (x + 1, y) + f (x − 1, y) + f (x, y + 1) + f (x, y − 1) − 4f (x, y)] (∗ ∗ ∗)

for all (x, y). (which is a linear system)

The above linear system can be solved by direct method (with big matrix) or by iterative method.
Simple iterative scheme: Let g be an N × N image.
Step 1: Let f 0 (x, y) = g(x, y) (Initial guess of the solution)

9
Step 2: For n ≥ 0 and for all (x, y), x = 1, · · · , M, y = 1, · · · , N ,

f n+1 (x, y) = g(x, y) + [f n (x + 1, y) + f n (x − 1, y) + f n (x, y + 1) + f n (x, y − 1) − 4f n (x, y)]

Impose boundary conditions by reflection:

f n+1 (0, y) = f n+1 (2, y); f n+1 (M + 1, y) = f n+1 (M − 1, y) for y = 1, · · · , N

n+1 n+1 n+1 n+1
f (x, 0) = f (x, 2); f (x, N + 1) = f (x, N − 1) for x = 1, · · · , M
n+1 n+1 n+1 n+1
f (0, 0) = f (2, 2); f (0, N + 1) = f (2, N − 1)
n+1 n+1 n+1
f (M + 1, 0) = f (M − 1, 2); f (M + 1, N + 1) = f n+1 (M − 1, N − 1)

(similar for f n )
Step 3: Continue the process until ||f n+1 −f n || ≤ tolerance. (Convergence depends on the spectral
radius of a matrix)
In fact, the solution of (***) is a minimizer of an energy (taking into account the extension by
reflection outside the domain):
M X
X N M X
X N
Ediscrete (f ) = (f (x, y) − g(x, y))2 + [(f (x + 1, y) − f (x, y))2 + (f (x, y + 1) − f (x, y))2 ]
x=1 y=1 x=1 y=1

Suppose f is a minimizer of Ediscrete . Then, for each position (x, y),

∂Ediscrete
0=
∂f (x, y)
= 2(f (x, y) − g(x, y)) + 2(f (x + 1, y) − f (x, y))(−1) + 2(f (x, y + 1) − f (x, y))(−1)
+ 2(f (x, y) − f (x − 1, y)) + 2(f (x, y) − f (x, y − 1))

After rearrangement,

f (x, y) = g(x, y) + [f (x + 1, y) + f (x − 1, y) + f (x, y + 1) + f (x, y − 1) − 4f (x, y)]

The continuous version of E(f ) is:

ZZ ZZ  2  2
2 ∂f ∂f
E(f ) = (f (x, y) − g(x, y)) dx dy + + dx dy
∂x ∂y
or ZZ ZZ
2
E(f ) = (f (x, y) − g(x, y)) dx dy + |∇f |2 dx dy

Remark:

• Solving −∆f + f = g is equivalent to solving an energy minimization problem.

• First term aims to find a smooth image, which is close enough to g (original image).
• Second term is called the regularization term, which is to minimize the derivative of f (En-
hance smoothness).
• −∇f + f = g can be solved in the frequency domain

DF T (f ) = DF T (g + ∆f ) = DF T (g + l ∗ f )
⇔ DF T (f )(u, v) = DF T (g)(u, v) + cDF T (l)(u, v)DF T (f )(u, v)
1
⇔ DF T (f )(u, v) = DF T (g)(u, v)
1 − cDF T (l)(u, v)

The inverse Fourier transform can then be applied.

10
1.5 Image denoising by solving PDE

Consider the harmonic-L2 image denoising model:

Z Z
E(f ) = (f (x, y) − g(x, y))2 dx dy + |∇f |2 dx dy
Ω

We find f that minimizes E(f ). Take any function v(x, y), and consider

s(ε) = E(f + εv)

Z Z
= (f (x, y) + εv(x, y) − g(x, y))2 dx dy + |∇f + ε∇v|2 dx dy
Ω Ω
Z
d
s(ε) = 2 (f (x, y) + εv(x, y) − g(x, y))v(x, y) dx dy
dε
Z Ω     
∂f ∂v ∂v ∂f ∂v ∂v
+2 +ε + +ε dx dy
Ω ∂x ∂x ∂x ∂y ∂y ∂y

d
If f is the minimizer, then s(ε) = 0
dε ε=0
Z Z
∴ s0 (0) = 0 = 2 (f (x, y) − g(x, y))v(x, y) dx dy + 2 (fx vx + fy vy ) dx dy
Ω Ω

Recall: (Integration by parts)

Z Z Z
∇f · ∇g dx dy = − (∇ · ∇f )g dx dy + g(∇f · ~n) ds
Ω Ω ∂Ω

where ~n = (n1 , n2 ) = outward normal on the boundary.

Z Z Z
∴ 0 = (f − g)v dx dy − (∇ · ∇f )v dx dy − (∇f · ~n)v ds
Ω Ω ∂Ω

Overall, we get Z Z
(f − g − ∆f )v dx dy − (∇f · ~n)v ds = 0
Ω ∂Ω
from where we obtain: (
f − g − ∆f = 0 in Ω
(PDE)
∇f · ~n = 0 on ∂Ω

Conversely, given f such that the above PDE is satisfied, for any other h
Z
E(h) − E(f ) = [(h − g)2 − (f − g)2 + |∇h|2 − |∇f |2 ] dx dy
Ω
Z
= [(h − g) − (f − g)]2 + |∇h − ∇f |2 + 2∇f · (∇h − ∇f ) + 2(f − g)(h − f )] dx dy
ZΩ
≥ 2∇f · ∇(h − f ) + 2(f − g)(h − f ) dx dy
Ω
Z Z
=2 − (∇ · ∇f )(h − f ) + (f − g)(h − f ) dx dy + 2 ∇f·
( : 0)(h − f ) = 0
~n

Ω | {z } ∂Ω
=0

Therefore, E(h) ≥ E(f ) and f is the minimizer.

Remark. • Energy minimization is usually done by:
1. Iterative method / Gradient descent

11
 2. Solving PDE

• Anisotropic diffusion is related to minimizing:

Z
E(f ) = K(x, y)|∇f (x, y)|2 dx dy
Ω

Intuitively, we want to minimize “derivative” / “jump” |∇f (x, y)|. On edges, K(x, y) is
small, so contribution of |∇f (x, y)| on the edges is small and thus less minimization of
|∇f (x, y)| on the edges.

• Energy minimization approaches are commonly called variational image denoising.

1.6 Total variation (TV) denoising

Other name: ROF model: Rudin, Osher, Fatemi

Motivation: The previous model solves f = g + ∆f , given a noisy image g. ∆f may smooth
out edges. Therefore, we consider ∇ · (K(x, y)f (x, y)) where K(x, y) is small if (x, y) is on the
boundary/edges.
Goal: Given a noisy image g(x, y), we look for f (x, y) that solves:
   
∂ 1 ∂f ∂ 1 ∂f
f =g+ + (∗ ∗ ∗∗)
∂x |∇f |(x, y) ∂x ∂y |∇f |(x, y) ∂y

Remark. Again, a problem arises when |∇f |(x, y) = 0. We will take care of it later.

We will show that the PDE (****) must be satisfied by a minimizer of the following energy
functional: ZZ ZZ
1 2
J(f ) = (f (x, y) − g(x, y)) dx dy + λ |∇f |(x, y) dx dy
2 Ω Ω
where Ω is the image domain.
∂J
To do this, we must have “ = 0”. Let’s discretize J(f ). The discrete version of J(f ) is
∂f
N N N X N p
1 XX X
J(f ) = (f (x, y) − g(x, y))2 + λ (f (x + 1, y) − f (x, y))2 + (f (x, y + 1) − f (x, y))2
2 x=1 y=1 x=1 y=1

∂f ∂f
(Finite difference approximation of and with ∆x = ∆y = 1)
∂x ∂y
Therefore,

∂J 2(f (x + 1, y) − f (x, y))(−1) + 2(f (x, y + 1) − f (x, y))(−1)

= (f (x, y) − g(x, y)) + λ p
∂f (x, y) 2 (f (x + 1, y) − f (x, y))2 + (f (x, y + 1) − f (x, y))2
2(f (x, y) − f (x − 1, y))
+λ p
2 (f (x, y) − f (x − 1, y))2 + (f (x − 1, y + 1) − f (x − 1, y))2
2(f (x, y) − f (x, y − 1))
+λ p =0
2 (f (x + 1, y − 1) − f (x, y − 1))2 + (f (x, y) − f (x, y − 1))2

12
By simplification, we get:
(
f (x + 1, y) − f (x, y)
f (x, y) − g(x, y) = λ p
(f (x + 1, y) − f (x, y))2 + (f (x, y + 1) − f (x, y))2
)
f (x, y) − f (x − 1, y)
− p
(f (x, y) − f (x − 1, y))2 + (f (x − 1, y + 1) − f (x − 1, y))2
(
f (x, y + 1) − f (x, y)
+λ p
(f (x + 1, y) − f (x, y))2 + (f (x, y + 1) − f (x, y))2
)
f (x, y) − f (x, y − 1)
− p
(f (x + 1, y − 1) − f (x, y − 1))2 + (f (x, y) − f (x, y − 1))2

for all 1 ≤ x, y ≤ N .
 
∇f
(Discretization of f − g = λ∇ · .)
|∇f |
The above equation is a non-linear equation. (Difficult to solve!!) We apply the gradient descent
method to solve it iteratively.
General gradient descent method
We consider the general case to solve:
minf J(f )
where f can be a number, a vector or a function. In our case, f is a discrete image function
(x, y) 7→ f (x, y). Hence, we consider J to be a function defined on a discrete image, which depends
on M × N variables (assuming the image is of size M × N ).
Consider a time-dependent image f (x, y; t). Assume f (x, y; t) solve the ODE:

df (x, y; t)
= −∇J(f (x, y; t)) (gradient descent equation)
dt

We show that J(f (x, y; t)) is decreasing as t increases.

In fact,
d
J(f (x, y; t)) = ∇J(f (x, y; t))f 0 (x, y; t) = −|∇J(f (x, y; t))|2 ≤ 0
dt

Therefore, J(f (x, y; t)) is decreasing as t increases. We solve (*) and f (x, y; t) → f¯(x, y) is the
minimizer.
We solve the gradient descent equation iteratively. Let f 0 (x, y) be the initial guess. (Here 1 ≤ x ≤
M, 1 ≤ y ≤ N and extend f 0 by reflection.)
We solve (*) in a discrete sense:

f n+1 − f n
= −∇J(f n ) (∗ ∗ ∗ ∗ ∗)
∆t

∆t is called the time step, which must be chosen carefully.

13
In our case,. (*****) becomes:

f n+1 (x, y) − f n (x, y)

∆t
f n (x + 1, y) − f n (x, y)
= −(f n (x, y) − g(x, y)) + λ p
(f n (x + 1, y) − f n (x, y))2 + (f n (x, y + 1) − f n (x, y))2
f (x, y) − f n (x − 1, y)
n
− λp
(f n (x, y) − f n (x − 1, y))2 + (f n (x − 1, y + 1) − f n (x − 1, y))2
f n (x, y + 1) − f n (x, y)
+ λp
(f n (x + 1, y) − f n (x, y))2 + (f n (x, y + 1) − f n (x, y))2
f n (x, y) − f n (x, y − 1)
− λp
(f n (x + 1, y − 1) − f n (x, y − 1))2 + (f n (x, y) − f n (x, y − 1))2

subject to the boundary condition imposed by reflection.

In the continuous setting, it is equivalent to:
 
df ∇f
= −(f − g) + λ∇ ·
dt |∇f |

Below is an example of TV/ROF denoising on an image:

2 Image deblurring in the spatial domain (Optional)

(Spatial) De-blurring de-noising variational model
Let g = h ∗ f + n be a blurred image, where h is the degradation functional and n is the noise.
Assume f is the original clean image, which is continuous on R2 . We consider the variational
model to find f that minimizes:
ZZ ZZ
1 2
J(f ) = (h ∗ f (x, y) − g(x, y)) dx dy + λ |∇f | dx dy
2 R2 R2

Remark. It is the TV de-noising model with image blur incorporated. We proceed to solve the
optimization problem by steepest (gradient) descent method.
Z ∞Z ∞
Observation: Let Hf (x, y) := h ∗ f (x, y) = h(α, β)f (x − α, y − β) dx dy.
Z ∞Z ∞ −∞ −∞

Let hf, gi = f (x, y)g(x, y) dx dy. Then:

−∞ −∞

hHf, gi = hf, H ∗ gi

14
where H ∗ is the adjoint of H. We want to compute H ∗ .
Z ∞Z ∞
hHf, gi = h ∗ f (x, y)g(x, y) dx dy
−∞ −∞
Z ∞Z ∞ Z ∞ Z ∞ 
= h(α, β)f (x − α, y − β) dα dβ g(x, y) dx dy
−∞ −∞ −∞ −∞
Z ∞ Z ∞ Z ∞ Z ∞ 
= f (x − α, y − β)g(x, y) dx dy h(α, β) dα dβ
−∞ −∞ −∞ −∞

Let X = x − α, Y = y − β, we obtain
Z ∞ Z ∞ Z ∞ Z ∞ 
hHf, gi = f (X, Y )g(X + α, Y + β) dX dY h(α, β) dα dβ
−∞ −∞ −∞ −∞
Z ∞Z ∞ Z ∞ Z ∞ 
= f (X, Y ) h(α, β)g(α + X, β + Y ) dα dβ dX dY
−∞ −∞ −∞ −∞

Let a = −α, b = −β, we obtain

Z ∞Z ∞ Z ∞ Z ∞ 
hHf, gi = f (X, Y ) h(−a, −b)g(X − a, Y − b)(−1)(−1) da db dX dY
−∞ −∞ −∞ −∞
Z ∞Z ∞ Z ∞ Z ∞ 
= f (X, Y ) h̃(a, b)g(X − a, Y − b) da db dX dY
−∞ −∞ −∞ −∞

where h̃(a, b) = h(−a, −b). Hence,

Z ∞Z ∞
hHf, gi = f (X, Y )h̃ ∗ g(X, Y ) dX dY = hf, H ∗ gi.
−∞ −∞

Therefore, H ∗ g(x, y) = h̃ ∗ g(x, y) .

Now, gradient descent algorithm can be done as follows.
d
Let s(ε) = J(f + εw), then s(ε) = 0. Then:
dε ε=0
ZZ ZZ
0 1 d 2 d
s (0) = (h ∗ f + εh ∗ w − g) dx dy + λ |∇f + ε∇w| dx dy
2 dε ε=0 dε ε=0
Z ∞Z ∞ ZZ  
1
= (h ∗ f − g)h ∗ w dx dy − λ ∇· ∇f w dx dy
−∞ −∞ |∇f |
Z ∞Z ∞  
1
= h̃ ∗ (h ∗ f − g) − λ∇ · ∇f w dx dy
−∞ −∞ |∇f |

In order to minimize J, the descent direction should be:

 
1
w = −h̃ ∗ (h ∗ f − g) + λ∇ · ∇f
|∇f |

Hence, the gradient descent algorithm is given by:

f n+1 − f n
 
1
= −h̃ ∗ (h ∗ f n − g) + λ∇ · ∇f n
dt |∇f n |
 
1 n
with dt the time step and suitable parameter added to ∇ · ∇f to avoid singularity.
|∇f n |
Remark. 1. Here, we consider the continuous image to be defined on the whole domain R2 .
2. In practice, if we are given an image defined on a compact domain Ω, we extend the image
to the whole R2 by setting the region outside Ω to be zero.

15
 3. Let TV-deblurring-denoising model simultaneously deblur an image and also denoise the im-
age, which preserves edges.
Further remark: In the discrete case, we may consider finding f (x, y) (0 ≤ x, y ≤ N − 1) which
minimizes:
N
X −1 N
X −1 N
X −1 N
X −1
E(f (x, y)) = (h ∗ f (x, y) − g(x, y))2 + α |∇f (x, y)|
x=0 y=0 x=0 y=0

We can again derive the gradient descent iterative scheme to minimize E(f ).

Again, we can observe the following.

N
X −1 N
X −1
Let Hf = h ∗ f . Define: hf, gi = f (x, y)g(x, y).
x=0 y=0
Then, hH(f ), gi = hf, H ∗ gi where H ∗ is the adjoint of H.
Similarly, we can find H T as follows.
N
X −1 N
X −1 N
X −1 N
X −1
hHf, gi = h(x − m, y − n)f (m, n)g(x, y)
x=0 y=0 m=0 n=0
N
X −1 N
X −1 N
X −1 N
X −1
= h(x − m, y − n)g(x, y)f (m, n)
m=0 n=0 x=0 y=0
−1 N −1 −1 N −1
N N
!
X X X X
= h̃(m − x, n − y)g(x, y) f (m, n)
m=0 n=0 x=0 y=0

where h̃(a, b) = h(−a, −b).

N
X −1 N
X −1
∴ hHf, gi = f (m, n)h̃ ∗ g(m, n)
m=0 n=0

Below is an example of TV/ROF deblurring on an image (with different values of λ):

16
3 Image sharpening in the spatial domain (Optional)
The image sharpening in the spatial domain is exactly the same as that in the frequency domain.
Instead of working on the frequency domain, we work on the spatial domain directly.
Let f be an input image. To sharpen the image, we compute a smoother image (by Gaussian
filtering or mean filtering) fsmooth . Define the sharper image g as:

g(x, y) = f (x, y) + k(f (x, y) − fsmooth (x, y))

When k = 1, the method is called unsharp masking.

When k > 1, the method is called highboost filtering.

Exercises
The entries of all matrices below are indexed by {0, 1, · · · , M − 1} × {0, 1, · · · , N − 1} unless
otherwise specified.

1. Apply the following filters:

i. the 3 × 3 mean filter;
ii. the 3 × 3 median filter;
 
4 1 0 1
1 0 0 0
iii. convolution filter 81 
0

0 0 0
1 0 0 0
on the following periodically extended 4 × 4 images:
 
1 0 0 0
0 1 0 0
(a) 
0 0 1
;
0
0 0 0 1
 
1 1 0 1
0 1 1 1
(b) 
1 1 1
;
0
1 0 1 1
 
1 −1 1 −1
−1 1 −1 1 
(c) 
 1 −1
.
1 −1
−1 1 −1 1
2. Find gi ∈ MN ×N (R), N ≥ 5, such that gi ∗ f = hi ∗ (hi ∗ f ) for any f ∈ MN ×N (R):

1
 if (x, y) = (−1, 0),
(a) h1 (x, y) = −1 if (x, y) = (0, 0),

0 otherwise;

(
1
9 if D(x, y) ≤ 2,
(b) h2 (x, y) =
0 otherwise;

1
2
 if D(x, y) = 0,
1
(c) h3 (x, y) = if D(x, y) = 1,
4
0 otherwise.


3. For the following energy functionals,

i. Find a necessary condition for optimality;

17
 ii. Verify whether the necessary condition is sufficient to guarantee optimality;
iii. Derive an iterative scheme to minimize the functional;
iv. Discretize the energy functional (with forward difference for first derivatives and central
difference for second derivatives);
v. Find a necessary condition for optimality for the discretized functional;
vi. Derive an iterative scheme to minimize the discretized functional.
Z
 
(a) E1 (f ) = (f − g)2 + λk∇f k2 dx dy, λ > 0;
ZΩ
(b) E2 (f ) = [(f − g)2 + Kk∇f k4 ] dx dy, K(x, y) > 0 non-constant;
ZΩ
(c) E3 (f ) = [(h ∗ f − g)2 + Kk∇f k] dx dy, K(x, y) > 0 non-constant.
Ω

4. Derive the anisotropic diffusion equation:

∂I(x, y; σ)
= ∇ · (K(x, y)∇I(x, y; σ))
∂σ
from minimizing the following energy functional:
Z
E(I) = K(x, y)k∇I(x, y)k2 dx dy.
Ω

18