Image and Video Analytics Unit II

UNIT II IMAGE PRE-PROCESSING

Local pre-processing - Image smoothing - Edge detectors - Zero-crossings of the second

derivative - Scale in image processing - Canny edge detection - Parametric edge models -
Edges in multi-speralct images - Local pre-processing in the frequency domain - Line
detection by local pre-processing operators - Image restoration.

2.1. Local pre-processing:

Local pre-processing methods are divided into two groups according to the goal of the
processing.
Smoothing aims to suppress noise or other small fluctuations in the image; it is
equivalent to the suppression of high frequencies in the Fourier transform domain.
Unfortunately, smoothing also blurs all sharp edges that bear important information
about the image.
Gradient operators are based on local derivatives of the image function. Derivatives
are bigger at locations of the image where the image function under goes rapid changes,
and the aim of gradient operators is to indicate such locations in the image.
Gradient operators have a similar effect to suppressing low frequencies in the Fourier
transform domain.

Noise is often high frequency in nature; if a gradient operator is applied to an image, the
noise level increases simultaneously. Clearly, smoothing and gradient operators have
conflict in gaims. Some pre-processing algorithms solve this problem and permit
smoothing and edge enhancement simultaneously.

Another classification of local pre-processing methods is according to the

transformation properties; linear and non-linear transformations can be
distinguished.
Linear operations calculate the resulting value in the output image pixel. lf(i,j)as a linear
combination of brightnesses in a local neighborhood of the pixel g(i, j) in the input
image. The contribution of the pixels in the neighborhood O is weighted by coefficients
h:

(5.23)
Equation(5.23)is equivalent to discrete convolution with the kernel h,which is called a
convolution mask. Rectangular neighbor hoods are often used with an odd number of
pixels in rows and columns, enabling specification of the central pixel of the neighbor-
hood.
The choice of the local transformation, size, and shape of the neighborhood depends
strongly on the size of objects in the processed image. If objects are rather large, an
image can be enhanced by smoothing of small degradations.
2.2 Image smoothing:
Image smoothing uses redundancy in image data to suppress noise, usually by some
form of averaging of brightness values in some neighborhood . Smoothing poses the
problem of blurring sharp edges, and so we shall consider smoothing methods which
are edge preserving here, the average is computed only from points in the
neighborhood which have similar properties to the point being processed.
Local image smoothing can effectively eliminate impulse noise or
degradationsappearingasthinstripes,butdoesnotworkifdegradationsarelarge blobs or
thick stripes. Such problems may be addressed by image restoration techniques.

Averaging, statistical principles of noise suppression

Assume that the noise value ν at each pixel is an independent random variable with
zero mean and standard deviation σ. We might capture the same static scene under the
same conditions n times. From each captured image a particular pixel value gi,i=1,...,n is
selected. An estimate of the correct value can be obtained as an average of these values,
with corresponding noise values ν1, . . . , νn

The second term here describes the noise, which is again ar and om value with zero
mean and standard deviation Thus, if n images of the same scene are available,
smoothing can be accomplished without blurring the image by

This reasoning is a well-known statistical result: a random sample is taken from a

population and the corresponding sample mean value is calculated. If
randomsamplesarerepeatedlyselectedandtheirsamplemeanvaluescalculated, we would
obtain a distribution of sample mean values. This distribution of sample means has
some useful properties:
• The mean value of the distribution of sample mean values is equal to the
population mean.
• Thedistributionofsamplemeanvalueshasvariance ,whichisclearly
smaller than that of than the original population.
• If the original distribution is normal (Gaussian) then the distribution of
samplemeanvaluesisalsonormal.Better,thedistributionofsamplemeans

converges to normal whatever the original distribution.This is the central limit

theorem.
• From the practical point of view, it is important that not too many random
se- lections have to be made. The central limit theorem tell us the distribution of sample
mean values without the need to create them. In statistics, usually about
30 samples are considered the lowest limit of the necessary number of observations.
Usually, only one noise corrupted is available, and averaging is then performed in a local
neighborhood. Results areacceptable if the noise is smaller in size than the smallest
objects of interest in the image, but blurring of edges is a
seriousdisadvantage.Averagingisaspecialcaseofdiscreteconvolution[equation
(5.23)].×Fora3x3 neighborhood,theconvolution maskhis

The significance of the pixel in the center of the convolution mask h or its 4neighbors is
sometimes increased, as it better approximates the properties of noise with a Gaussian
probability distribution.

There are two commonly used smoothing filters whose coefficients gradually decrease
to have near-zero values at the window edges. This is the best way to minimize spurious
oscillations in the frequency spectrum. These are the Gaussian and the Butter worth
filters. Larger convolution masks for averaging by Gaussian filter are created according
to the Gaussian distribution formula (equation 5.47) and the mask coefficients are
normalized to have a unit sum.

(a) (b)
(c) (d)
Figure 5.9: Noise with Gaussian distribution and averaging filters. (a) Original image. (b)
Superimposed noise (random Gaussian noise characterized
byzeromeanandstandarddeviationequaltoone-halfofthegray-levelstandard deviation of
the original image). (c) 3 × 3 averaging. (d) 7 × 7 averaging.

An example will illustrate the effect of this noise suppression(lower solution images,
256 256, were chosen deliberately to show the discrete nature of the process). Figure
5.9a shows an original image of Prague castle; Figure 5.9b shows the same image with
superimposed additive noise with Gaussian distribution; Figure 5.9c shows the result of
averaging with a 3x 3 convolution mask (equation 5.27)— noise is significantly reduced
and the image is slightly blurred. Averaging with a larger mask (7 x 7) is demonstrated
in Figure 5.9d, where the blurring is much more serious.
Such filters can be very computationally costly, but this is considerably reduced in the
important special case of separable filters. Separability in 2D means that the
convolution kernel can be factorized as a product of two one- dimensional vectors, and
theory provides a clue as to which convolution masks are separable.
Asanexample, consider abinomic filter. Its elements are binomic numbers which are
created as a sum of the corresponding two numbers in
Pascal’striangle.Considersuchafilterofsize5x5—itcanbedecomposedintoaproduct
Of two
1Dvectors,h1,h2.

Supposeaconvolutionkernelisofsize2N+1.Equation(5.23)allows theconvolution to
beRewrittentaking account ofthespecial properties of separability

Thedirectcalculationoftheconvolutionaccordingtoequation(5.23)would need, in our

case of 5 5 convolution kernel, 25 multiplications and 24 additions for each pixel. If the
separable filter is used then only 10 multiplications and 8 additions suffice.

Averaging with limited data validity

Methods that average with limited data validity try to avoid blurring by
averagingonlythosepixelswhichsatisfysomecriterion,theaimbeingtoprevent involving
pixels that are part of a separate feature.
Averysimplecriterionistodefineabrightnessintervalofinvaliddata[min, max]
(typicallycorrespondingtonoiseofknownimagefaults),andapplyimage
averagingonlytopixelsinthatinterval.Forapoint(m,n),theconvolutionmask is calculated
in the neighborhood O by the non-linear formula

where (i, j) specify the mask element. Therefore, only values of pixels with invalidgray-
levelsarereplacedwithanaverageoftheirneighborhoods,andonly valid data contribute to
the averages.
Asecondmethodperformsaveragingonlyifthecomputedbrightnesschange ofapixel is in
somepre-defined interval; this permits repairto large-area errors resulting from slowly
changing brightness of the background without affecting
therestoftheimage.Athirdmethodusesedgestrength(i.e.,gradientmagnitude)
asacriterion.Themagnitudeofsomegradientoperatorisfirstcomputedforthe entire image,
and only pixels with a small gradient are used in averaging. This method effectively
rejects averaging at edges and thereforesuppresses blurring, but setting of the threshold
is laborious.

(a) (b)
Figure5.10: Averagingwithlimiteddatavalidity.(a)Originalcorrupted image. (b) Result of
corruption removal.

Averagingaccordingtoinversegradient
Withinaconvolutionmaskofoddsize,theinversegradientδofapoint(i,j) with respect to the
central pixel (m, n) is defined as
Ifg(m,n)=g(i,j),then wedefineδ(i,j)=2,soδisintheinterval(0,2],and is smaller at the edge
than in the interior of a homogeneous region. Weight coefficients in the convolution
mask h are normalized by the inverse gradient, and thewholeterm is multiplied by 0.5
tokeep brightness valuesin theoriginal
range:themaskcoefficientcorrespondingtothecentralpixelisdefinedash(i,j)
= 0.5. The constant 0.5 has the effect of assigning half the weight to the central pixel (m,
n), and the other half to its neighborhood

Thismethodassumessharpedges.Whentheconvolutionmaskisclosetoanedge, pixels from

the region have larger coefficients than pixels near the edge, and it is not blurred.
Isolated noise points within homogeneous regions have small values of the inverse
gradient; points from the neighborhood take part in averaging and the noise is removed.
Averagingusingarotatingmask
Thesmoothingdiscussedthusfarwaslinear,whichhasthedisadvantagethat edges in the
image are inevitably blurred.Alternative non-linear methods exist
whichreducethis.Theneighborhoodofthecurrentpixelisinspectedanddivided
intotwosubsetsbyahomogeneitycriterionoftheuser’schoice.Onesetconsists of all pixels
neighboring the current pixel or any pixel already included in this set, which satisfy the
homogeneity criterion.The second set is the complement. Thisselectionoperationisnon-
linearandcausesthewholefiltertobenon-linear. Having selected the homogeneous subset
containing the current pixel, the most probable value is sought in it by a linear or non-
linear technique.

Averagingusingarotatingmaskissuchanon-linearmethodthatavoidsedge blur-
ring,andtheresultingimageisinfactsharpened.Thebrightnessaverageis
calculatedonlywithinthisregion;abrightnessdispersionσ2isusedastheregion
homogeneity measure. Let n be the number of pixels in a region R and g be the input
image. Dispersion σ2 is calculated as

Havingcomputedregionhomogeneity,weconsideritsshapeandsize.Theeight possible 3x3

masks that cover a 5 x 5 neighborhood of a current pixel (marked by
thesmallcross)areshowninFigure5.11.Theninthmaskisthe3x3neighborhood
ofthecurrentpixelitself.Othermaskshapes—largerorsmaller—canalsobe used.

Figure5.11:Eightpossiblerotated3×3 masks.

Algorithm5.2:Smoothingusingarotatingmask
1. Considereachimagepixel(i,j).
2. Calculatedispersionforallpossiblemaskrotationsaboutpixel(i,j)according to
equation (5.31).
3. Choosethe mask with minimum dispersion.
4. Assign to the pixel f (i, j) in the output image f the average brightness in the
chosen mask.

Algorithm5.2canbeusediterativelyandtheprocessconvergesquitequickly to a stable
state. The size and shape of masks influence the convergence—the smaller the mask,
the smaller are the changes and more iterations are needed.A larger
mas ksuppressesnoisefaster andthesharpeningeffectisstronger.Onthe other hand,
infor mation about details smaller than the mask may be lost. The number of iterations
is als o influenced by the shape of regions in the image and noise properties.

Med ianfiltering
In probability theory, the median divides the higher half of a probability dist ribution
fromthe lower half. For a random variable x, the median M is the value for w hich the
probability of the outcome x < M is 0.5. The median of a finite list of real num bers is
simply found by ordering the list and selecting the middle member. Lists a re often
cons tructed to be odd in length to secure uniqueness.

Medi an filtering is a non-linear smoothing method that reduces the blurring of e dges, in
which the idea is to replace the current point in the image by the
medianofthebrightnessesinitsneighborhood.Themedianintheneighborhood is not
affected by individual noise spikes and so median smoothing eliminates impul se noise
quitewell.Further, as median filtering does not blur edges much, it can be applied
iteratively. Clearly, performing a sort on pixels within a (possibly large) rect angular
win dow at every pixel position may become very expensive. A more efficient a pproach
[Huang et al., 1979; Pitas and Venetsanopoulos, 1990] is to notice that as the window
moves across a row by one column, the only change to its
–con tents is to lose theleftmost column and replace it with a new rightco lumn—
foram edianwindowofmrowsandncolumns,mn2mpixelsare unchanged and do n ot need
re-sorting. The algorithm is as follows:

Downloaded by J.Priya ,Asst.Prof - AI & DS Dept ([email protected])

 7. (Wehave nm>t,ifhere).Repeat

nm=nm−H[m],m

=m−1,

untilnm≤t.

8. Iftheright-handcolumnofthewindowisnotattheright-handedgeofthe image,
go to (3).

9. Ifthebottomrowofthewindowisnotatthebottomoftheimage,goto (2).

Median filtering is illustrated in Figure 5.12. The main disadvantage of median filtering
in a rectangular neighborhood is its damaging of thin lines and sharpcorners—
thiscanbeavoidedifanothershapeofneighborhoodisused.For
instance,ifhorizontal/verticallinesneedpreserving,aneighborhoodsuchasthat in Figure
5.13 can be used.
Median smoothing is a special instance of more general rank filtering techniques, the
idea of which is to order pixels in some neighborhood into a sequence.Theresultsofpre-
processingaresomestatisticsoverthissequence,of which the median is one possibility.
Another variant is the maximum or the minimum values of the sequence. This defines
generalizations of dilation and erosion operators in images with more brightness
values.
(a) (b)
Figure 5.12: Median filtering. (a) Image corrupted with impulse noise (14% of image
area covered with bright and dark dots). (b) Result of 3x 3 median filtering.

Non-linearmeanfilter
Figure 5.13: Horizontal/vertical line preserving neighbor- hood for median filtering

Thenon-linearmeanfilterisanothergeneralizationofaveragingtechniques; it is defined by

wheref(m,n)istheresultofthefiltering,g(i,j)isthepixelintheinputimage,Oand is a local
neighborhood of the current pixel (m, n). The function u of one variable has an inverse
function u−1; the a(i, j) are weight coefficients.
Iftheweightsa(i,j)areconstant,thefilteriscalled homomorphic.Some homomorphic
filters used in image processing are:
• Arithmeticmean,u(g) = g.
• Harmonicmean, u(g)=1/g .
• Geometricmean,u(g) =logg .

2.3. Edgedetectors:
Edgedetectorsareacollectionofveryimportantlocalimagepre-processing
methodsusedtolocatechangesintheintensityfunction;edgesarepixels where brightness
changes abruptly.

 Edgesarethoseplacesinanimagethatcorrespondtoobject boundaries.
 Edgesarepixelswhereimagebrightnesschanges abruptly.

Neurological and psychophysical research suggests that locations in the image in which
the function value changes abruptly are important for image
perception.Edgesaretoacertaindegreeinvarianttochangesofilluminationand viewpoint. If
only edge elements with strong magnitude (edges) areconsidered, such information
often suffices for image understanding. The positive effect of
suchaprocessisthatitleadstosignificantreductionofimagedata.Nevertheless such data
reduction does not undermine understanding the content of the image (interpretation)
in many cases. Edge detection provides appropriate generalization of the image data;
for instance, line drawings perform such a generalization.
Weshallconsiderwhichphysicalphenomenaintheimageformationprocess lead to abrupt
changes in image values—see Figure 5.15. Calculus describes
changesofcontinuousfunctionsusingderivatives;animagefunctiondependson two
variables—co-ordinates in the image plane—and so operators describing
edgesareexpressedusingpartialderivatives.Achangeoftheimagefunctioncan
bedescribedbyagradientthatpointsinthedirectionofthelargestgrowthofthe image
function.

Figure 5.15: Origin of edges, i.e., physical phenomena in image formation process which
lead to edges in images.At right, a Canny edge detection.

An edge is a property attached to an individual pixel and is calculated from the image
function behavior in a neighborhood of that pixel. It is a vector variable with two
components, magnitude and direction.The edge magnitude is
themagnitudeofthegradient, and the edgedirection φ is rotatedwith respect to −
the gradient direction ψ by 90◦.The gradient direction gives the direction of
maximumgrowthofthefunction,e.g.,fromblackf(i,j)=0towhitef(i,j)=
255. This is illustrated in Figure 5.16, in which closed lines are lines of equal brightness.
Theorientation0◦points east.
Edges are often used in image analysis for finding region boundaries. Provided that the
region has homogeneous brightness, its boundary is at the pixels where the image
function varies and so in the ideal case without noise consists of pixels with high edge

Figure5.16: Gradientdirection andedgedirection.

magnitude. It can be seen that the boundary and its parts (edges) are perpendicular to
the direction of the gradient.
Figure5.17showsexamplesofseveralstandardedgeprofiles.Edgedetectors are usually
tuned for some type of edge profile.
g g g g

x x x x
Figure5.17:Typicaledgeprofiles.
The gradient magnitude | grad g(x, y)|and gradient direction ψ are continuous image
functions calculated as

wherearg(x, y) is the angle (in radians) from the x axis to (x, y). Sometimes
weareinterestedonlyinedgemagnitudeswithoutregardtotheirorientations— a linear
differential operator called the Laplacianmay then be used. The Laplacian has the same
properties in all directions and is therefore invariant to rotation. It is defined as
Image sharpening [Rosenfeld and Kak, 1982] has the objective of making edgessteeper
—thesharpenedimageisintendedtobeobservedbyahuman.The sharpened output image f
is obtained from the input image g as
f(i, j) =g(i, j)−C S(i, j), (5.36)
where C is a positive coefficient which gives the strength of sharpening and S(i, j) is a
measure of the image function sheerness, calculated using a gradient
operator.TheLaplacianisveryoftenusedforthispurpose.Figure5.18givesan example of
image sharpening using a Laplacian.
Image sharpening can be interpreted in the frequency domain as well. We know that the
result of the Fourier transform is a combination of harmonic functions.
Thederivativeof the harmonicfunction sin(nx) is n cos(nx);thus the higher the
frequency, the higher the magnitude of its derivative.
A similar image sharpening technique to that of equation (5.36), called unsharp
masking, is often used in printing industry applications. A signal
proportionaltoanunsharp(e.g.,heavilyblurredbyasmoothingoperator)image
issubtractedfromtheoriginalimage.Adigitalimageisdiscreteinnatureandso equations
(5.33) and (5.34), containing derivatives, must be approximated by
differences.Thefirstdifferencesoftheimagegintheverticaldirection(forfixed i) and in the
horizontal direction (for fixed j) are given by
∆ig(i, j) =g(i, j) −g(i − n, j) ,
∆jg(i, j) =g(i, j) −g(i, j−n) ,

(5.37)

where n is a small integer, usually 1. The value n should be chosen small enough to
provide a good approximation to the derivative, but large enough to neglect
unimportant changes in the image function. Symmetric expressions for the differences,
∆ig(i, j) =g(i +n,j) −g(i − n, j) ,
∆jg(i, j) =g(i, j + n) −g(i, j −n),
arenot usuallyused becausetheyneglect theimpactof thepixel(i, j) itself.

(a) (b)
Figure5.18:Laplacegradientoperator.(a)Laplaceedgeimageusingthe8-connectivity mask.
(b) SharpeningusingtheLaplaceoperatorequation5.36,C=0.7.Comparethe
sharpening effect with the original image in Figure 5.9a.

Gradient operators as a measure of edge sheerness can be divided into three categories:
1. Operators approximating derivatives of the image function using
differences. Some are rotationally invariant (e.g., Laplacian) and thus are
computedfromoneconvolutionmaskonly.Others,whichapproximatefirst derivatives, use
several masks.The orientation is estimated on the basis of the best matching of several
simple patterns.
2. Operatorsbasedonzero-crossingsoftheimagefunctionsecondderivative (e.g.,
Marr-Hildreth or Canny edge detectors).
3. Operatorswhichattempttomatchanimagefunctiontoaparametricmodel of
edges.

Edgedetectionisanextremelyimportantstepfacilitatinghigher-levelimage
analysisandremainsanareaofactiveresearch.Examplesofthevarietyof
approaches found in current literature are fuzzy logic, neural networks, or wavelets. It
may be difficult to select the most appropriate edge detection strategy.
Individual gradient operators that examine small local neighborhoods are in fact
convolutions (cf. equation 5.23), and can be expressed by convolution masks. Operators
which are able to detect edge direction are represented by a collection of masks, each
corresponding to a certain direction.
Robertsoperator

The Roberts operator is one of the oldest [Roberts, 1965], and is very easy tocompute as
it uses only a 2 × 2 neighborhood of the current pixel. Its masks are

- - -
sothemagnitude oftheedgeiscomputed asg(i, j) −g(i + 1, j + 1) + g(i, j
-
+1) −g(i+1 , j) . (5.40)
The primary disadvantage of the Roberts operator is its high sensitivity to noise,
because very few pixels are used to approximate the gradient.
Laplaceoperator
TheLaplaceoperator∇2isaverypopularoperatorapproximatingthesecond
derivativewhichgivestheedgemagnitudeonly.TheLaplacian,equation(5.35), is
×approximatedindigitalimagesbya convolution sum.A3x3 mask h is often used; for 4-
neighborhoods and 8- neighborhoods it is defined as

ALaplacian operator with stressed significanceof the central pixel or its neighborhood is
sometimes used. In this approximation it loses invariance to rotation

TheLaplacianoperatorhasadisadvantage—itrespondsdoublytosome edges in the image.

Prewittoperator
The Prewitt operator, similarly to the Sobel, Kirsch, and some other operators,
approximates the first derivative. The gradient is estimated in eight (for a 3x 3
convolution mask) possible directions, and the convolution result of greatest magnitude
indicates the gradient direction. Larger masks are possible. We present only the first
three 3 x 3 masks for each operator; the others can be created by simple rotation.

Thedirection ofthegradient is givenby themask giving maximal response. This is also the
case for all the following operators approximating the first derivative.

(a) (b)

(c) (d)
Figure 5.19: First-derivative edge detection using Prewitt operators. (a) North
direction(thebrighterthepixelvalue,thestrongertheedge).(b)Eastdirection.(c) Strong
edges from (a).
(d) Strongedgesfrom(b).
Sobel operator
The Sobel operator is often used as a simple detector of horizontality and verticality of
edges, in which case only masks h1and h3are used. If the h1response is y and the
h3response x, we might then derive edge strength (magnitude) as

Anddirectionasarctan(y/x).
Kirsch operator

To illustrate the application of gradient operators on real images, consider again the
image given in Figure 5.9a. The Laplace edge image calculated is
showninFigure5.18a;thevalueoftheoperatorhasbeenhistogramequalizedto enhance its
visibility.
The properties of an operator approximating the first derivative are
demonstratedusingthePrewittoperator—resultsofothersaresimilar.Theoriginal image is
again given in Figure 5.9a; Prewitt approximations to the directional gradients are in
Figures 5.19a,b, in which north and east directions are shown. Significant edges (those
with above-threshold magnitude) in the two directions are given in Figures 5.19c,d.
2.4. Zero-crossingsofthesecondderivative:
Inthe1970s,Marr’stheoryconcludedfromneurophysiologicalexperiments that
object boundaries are the most important cues that link an intensity image with its
interpretation. Edge detection techniques existing at that time (e.g., the Kirsch, Sobel,
and Pratt operators) were based on convolution in very small
neighborhoodsandworkedwellonlyforspecificimages.Themaindisadvantage of these
edge detectors is their dependence on the size of the object and sensitivity to noise.
An edge detection technique based on the zero-crossings of the second derivative
(Marr-Hildrethedge detector)explores the fact that a step edge
correspondstoanabruptchangeintheimagefunction.Thefirstderivativeofthe image
function should have an extremum at the position corresponding to the edge in the
image, and so the second derivative should be zero at the same position; however, it is
much easier and more precise to find a zero-crossing position than an extremum. In
Figure 5.20 this principle is illustrated in 1D for
thesakeofsimplicity.Figure5.20ashowsstepedgeprofilesoftheoriginalimage
functionwithtwodifferentslopes,Figure5.20bdepictsthefirstderivativeofthe
imagefunction,andFigure5.20cillustratesthesecondderivative;noticethatthis crosses the
zero level at the same position as the edge.
Considering a step-like edge in 2D, the 1D profile of Figure 5.20a corresponds to a cross
section through the 2D step. The steepness of the profile will change if the

Figure5.20: 1Dedgeprofileof thezero-crossing.

orientation of the cutting plane changes—the maximum steepness is observed when the
plane is perpendicular to the edge direction.
The crucial question is how to compute the second derivative robustly. One
possibilityistosmoothanimagefirst(toreducenoise)andthencomputesecond
derivatives.When choosingasmoothingfilter, therearetwo criteriathatshould be fulfilled.
First, the filter should be smooth and roughly band limited in the
frequencydomaintoreducethepossiblenumberoffrequenciesatwhichfunction
changescantakeplace.Second,theconstraintofspatiallocalizationrequiresthe responseof
afilterto be fromnearbypoints in the image.Thesetwo criteriaare conflicting, but they
can be optimized simultaneously using a Gaussian distribution. In practice, one has to
be more precise about what is meant by the localization performance of an operator,
and the Gaussian may turn out to be suboptimal. We shall consider this in the next
section.
The 2D Gaussian smoothing operator G(x, y) (also called a Gaussian filter, or simply a
Gaussian) is given by

where x, y are the image co-ordinates and σ is a standard deviation of the associated
probability distribution. Sometimes this is presented with a normalizing factor

ThestandarddeviationσistheonlyparameteroftheGaussianfilter—itis
proportionaltothesizeoftheneighborhoodonwhichthefilteroperates.Pixelsmore
distantfromthecenteroftheoperatorhavesmallerinfluence,andpixelsfartherthan 3σ from
the center have negligible influence.
Ourgoal is to obtain a second derivative of a smoothed 2D function f (x, y).
WehavealreadyseenthattheLaplaceoperatorgivesthesecondderivative,and is non-
directional (isotropic). Consider then the Laplacian of an image f (x, y)
smoothedbyaGaussian(expressedusingaconvolution).Theoperationisoften abbreviated
as LoG, from Laplacian of Gaussian

Theorderofperformingdifferentiationandconvolutioncanbeinterchanged because of the

linearity of the operators involved

ThederivativeoftheGaussian filter canbepre-computedanalytically,

since
it isindependent of the image under consideration, and so the complexityof the
composite operation is reduced. From equation (5.47), we see

andsimilarlyfory. Hence

Afterintroducinganormalizingmultiplicativecoefficientc,wegeta convolution mask of a

LoG operator:

where c normalizes the sum of mask elements to zero. Because of its shape, the
invertedLoGoperatoriscommonlycalledaMexicanhat.Anexampleofa5× 5 discrete
approximation (wherein a 17 × 17 mask is also given) is
Of course, these masks represent truncated and discrete representations of infinite
continuous functions, and care should be taken in avoiding errors in moving to this
representation.
Finding second derivatives in this way is very robust. Gaussian smoothing
effectivelysuppressestheinfluenceofthepixelsthataremorethanadistance3σ from the
current pixel; then the Laplace operator is an efficient and stable measure of changes in
the image.

Afterimageconvolution withthelocationsintheconvolvedimagewhere the zero

level is crossed correspond to the positions of edges. The advantage of this approach
compared to classical edge operators of small size is that a larger area
surroundingthecurrentpixelistakenintoaccount;theinfluenceofmoredistantpoints
decreasesaccordingtotheσoftheGaussian.Intheidealcaseofanisolatedstepedge, the σ
variation does not affect the location of the zero-crossing. Convolution masks become
large for larger σ; for

example, σ = 4 needs a mask about 40 pixels wide. Fortunately, there is a separable

decomposition of the operator that can speed up computation considerably.
The practical implication of Gaussian smoothing is that edges are found reliably.Ifonly
globallysignificant edgesarerequired,thestandarddeviation σ of the Gaussian smoothing
filter may be increased, having the effect of suppressing less significant evidence.

The operator can be very effectively approximated by convolution with a

maskthatis thedifferenceoftwo Gaussian averagingmaskswithsubstantiallydifferent σ—
this method is called the difference of Gaussians, abbreviated as DoG.
When implementing a zero-crossing edge detector, trying to detect zeros in the LoG or
DoG image will inevitably fail, while naive approaches of thresholding the
×
LoG/DoG image and defining the zero-crossings in some interval of values close to zero
give piece- wise disconnected edges at best. To create a well- functioning second-
derivative edge detector, it is necessary to implement a true zero-crossing detector. A
simple detector may identify a zero-crossing in a moving 2 × 2 window, assigning an
edge label to any one corner pixel, say the
upperleft,ifLoG/DoGimagevaluesofbothpolaritiesoccurinthe2x2window;
no edge label would be givenifvalueswithin the windoware either all positive
orallnegative.Anotherpost-processingsteptoavoiddetectionofzero-crossings
corresponding to non-significant edges in regions of almost constant gray-level
wouldadmitonlythosezero-crossingsforwhichthereissufficientedgeevidence from a first-
derivative edge detector. Figure 5.21 provides several examples of edge detection using
zero crossings of the second derivative.
Many other approaches improving zero-crossing performance can be found in the
literature; some of them are used in pre-processingor post-processing steps. The
traditional second-derivative zero-crossing technique has disadvantages as

(a) (b)

(c) (d)
Figure 5.21: Zero-crossings of the second derivative, see Figure 5.9a for the original
image.
(a) DoGimage(σ1=0.10,σ2=0.09),darkpixels correspondtonegativevalues, bright pixels
to positive. (b) Zero-crossings of the DoG image. (c)
DoGzerocrossingedgesafterremovingedgeslackingfirst-derivativesupport.(d) LoG zero-
crossing edges (σ = 0.20) after removing edges lacking first- derivative support—note
different scale of edges due to different Gaussian smoothing parameters.
well. First, it smooths the shape too much; for example, sharp corners are lost. Second, it
tends to create closed loops of edges (nicknamed the ‘plate of spaghetti’effect).
Neurophysiologicalexperimentsprovideevidencethatthehumaneyeretina
intheformoftheganglioncellsperformsoperationsverysimilartothe operations.Ea
chsuchcellrespondstolightstimuliinalocalneighborhoodcalled the receptive field, which
has a center-surround organization of two complementary types, off-center and on-
center. When a light stimulus occurs, activity of on-center cells increases and that of off-
center cells is inhibited. The retinal operation on the image can be described analytically
as the convolution of the image with the 2G operator.
2.5. Scaleinimageprocessing:
• Manyimageprocessingtechniquesworklocally,theoreticallyatthelevelof
individual pixels—edge detection methods are an example. The essential problem in
such computation is scale.
• Edges correspond to the gradient of the image function, which is computed as
a difference between pixels in some neighborhood.
• There is seldom a sound reason for choosing a particular size of
neighborhood, since the ‘right’size depends on the size of the objects under
investigation.
• To know what the objects are assumes that it is clear how to interpret an
image, and this is not in general known at the pre-processing stage.
• The solution to the problem formulated above is a special case of a general
paradigm called the system approach. This methodology is common in cybernetics or
general system theory to study complex phenomena.
• Thephenomenon underinvestigation is expressed at different resolutions of
the de- scription, and a formal model is created at each resolution. Then the
qualitativebehaviorofthemodelisstudiedunderchangingresolutionofthe description.
Such a methodology enables the deduction of meta-knowledge about the phenomenon
that is not seen at the individual description levels.
• Different description levels are easily interpreted as different scales in the
domain of digital images. The idea of scale is fundamental to Marr’s edge
detectiontechnique,wheredifferentscalesareprovidedbydifferentsizesof
Gaussianfiltermasks.Theaimwasnotonlytoeliminatefinescalenoisebut also to separate
events at different scales arising from distinct physical processes.
• Assumethatasignalhasbeensmoothedwithseveralmasksofvariablesizes.
Everysettingofthescaleparametersimpliesadifferentdescription,butitis
notknownwhichoneiscorrect;formanytasks,noonescaleiscategorically correct. If the
ambiguity introduced by the scale is inescapable, the goal of scale-
independentdescriptionistoreducethisambiguityasmuchaspossible.
• Hereweshallconsiderjustthreeexamplesoftheapplicationofmultiplescale
description to image analysis.

1. Thefirstapproachaimstoprocessplanarnoisycurvesatarangeof scales—thesegmentof
curvethatrepresentstheunderlyingstructureof
thesceneneedstobefound.Theproblemisillustratedbyanexampleof two noisy curves.
• One of these may be interpreted as a closed curve, while the other could be
described as two intersecting straight lines.
• Local tangent direction and curvature of the curve are significant only with
some idea of scale after the curve is smoothed by a Gaussian filter with varying
standard deviations.

Figure5.22:Curvesthat maybeanalyzedat multiple

scales.

2. Asecondapproach,calledscale-spacefiltering,triestodescribesignals
qualitatively with respect to scale. The problem was formulated for 1D
signalsf(x),butitcaneasilybegeneralizedfor2Dfunctionsasimages.
Theoriginal1Dsignalf(x)issmoothedbyconvolutionwitha1D
Gaussian

Ifthestandard deviationσisslowlychanged,thefunction

representsasurfaceonthe(x, σ)planethatiscalledthescale-space image.

Inflectionpointsof thecurveF (x, σ0)foradistinct value σ0

describethecurvef(x)qualitatively.Thepositionsofinflexionpointscanbe drawn as a set of

curves in (x, σ) co-ordinates. Coarse to fine analysis of the
curvescorrespondingtoinflexionpoints,i.e.,inthedirectionofdecreasingvalue of the σ,
localizes large-scale events.
The qualitative information contained in the scale-space image can be
transformedintoasimple intervaltreethatexpressesthestructureofthesignal f (x) over all
observed scales. The interval tree is built from the root that
correspondstothelargestscale(σmax),andthenthescale-spaceimageissearched
inthedirectionofdecreasingσ.Theintervaltreebranchesatthosepointswhere new curves
corresponding to inflexion points appear
3. The third example of the application of scale is that used by the popular
Canny edge detector. Since the Canny detector is a significant and widely used
contribution to edge detection techniques, its principles will be explained in detail.
2.6. Cannyedgedetection:
Cannyproposedanapproachtoedgedetectionthatisoptimalforstepedges
corruptedbywhitenoise.Theoptimalityofthedetectorisrelatedtothreecriteria.
1) Thedetectioncriterionexpressesthefactthatimportantedgesshould not
be missed and that there should be no spurious responses.
2) Thelocalizationcriterionsaysthatthedistancebetweentheactualand
located position of the edge should be minimal.
3) The one response criterion minimizes multiple responses to a single
edge.Thisispartlycoveredbythefirstcriterion,sincewhenthereare two responses to a
single edge, one of them should be considered as false. This third criterion solves the
problem of an edge corrupted by noise and works against non-smooth edge operators.

Canny’sderivationisbasedonseveralideas.
1. Theedgedetectorwasexpressedfora1Dsignalandthefirsttwooptimality
criteria.Aclosed-form solution wasfound using thecalculusof variations.
2. Ifthethirdcriterion(multipleresponses)isadded,thebestsolutionmaybe found
by numerical optimization.The resulting filter can be approximated effectively with
error less than 20% by the first derivative of a Gaussian smoothing filter with standard
deviation σ [Canny, 1986]; the reason for
doingthisistheexistenceofaneffectiveimplementation.Thereisastrong similarity here to
the LoG based Marr-Hildreth edge detector.
3. Thedetectoristhengeneralizedtotwodimensions.Astepedgeisgivenby
itsposition,orientation,andpossiblymagnitude(strength).Itcanbeshown that convolving
an image with a symmetric 2D Gaussian and then differentiating in the direction of the
gradient (perpendicular to the edge direction) forms a simple and effective directional
operator (recall that the Marr-Hildrethzerocrossing operator does not give information
about edge direction, as it uses a Laplacian filter).
Suppose G is a 2D Gaussian [equation (5.47)] and assume we wish to convolve the
image with an operator Gnwhich is a first derivative of G in some direction n

We would like n to be perpendicular to the edge: this direction is not known in advance,
but a robust estimate of it based on the smoothed
gradientdirectionisavailable.Iffistheimage,thenormaltotheedge nis estimated as
 Theedgelocationisthenatthelocalmaximumoftheimagefconvolved with the operator
Gnin the direction n

Substitutinginequation(5.56)forGnfromequation(5.54),weget

Thisequation(5.57)illustrateshowtofindlocalmaximainthedirection perpendicular to the

edge; this operation is often referred to as non- maximal suppression (see also
Algorithm 6.4).
Astheconvolutionandderivativeareassociativeoperationsinequation (5.57),wecan first
convolvean imagef with a symmetricGaussian G and then compute the directional
second-derivative using an estimate of the
directionncomputedaccordingtoequation(5.55).Thestrengthoftheedge (magnitude of
the gradient of the image intensity function f ) is measured as

4. Spurious responses to a single edge caused by noise usually create a ‘streaking’problem

that is very common in edge detection in general.The
outputofanedgedetectorisusuallythresholdedtodecidewhichedgesare significant, and
streaking may break up edge contours as the operator fluctuates above and below the
threshold. Streaking can be eliminated by thresholding with

Downloaded by J.Priya ,Asst.Prof - AI & DS Dept ([email protected])

Algorithm5.4:Cannyedgedetector
ConvolveanimagefwithaGaussianofscaleσ.

Estimate localedge normal directions n using equation(5.55) for each pixelin the image.

Findthelocation oftheedgesusingequation (5.57) (non-maximal suppression).

Computethemagnitudeoftheedgeusingequation(5.58).

Thresholdedgesintheimagewithhysteresistoeliminatespuriousresponses.
hysteresis, employing a hard (high) threshold and a soft (lower) threshold— see
Re pe a ts t e p s (1) th ro u g h( 5 ) forasc en d in g val u e sof t h est a nd a rdd e v ia t io n σ .
A lg o r i t h m 6 .5 . Th e l o w an d h ig h th r e s h o l ds are s e t ac co r ding to an estimated signal-to-
A g gnr eogi as tee trhaetf ii no a. linformationaboutedgesatmultiplescaleusingthe‘feature synthesis’
5. Ta hp ep r o ca oc hr . r e c t scale for the operator depends on the objects contained in the
image.Thesolutiontothisunknownistousemultiplescalesandaggregate information from
them. Different scales for the Canny detector are
representedbydifferentstandarddeviationsσoftheGaussians.Theremay be several scales
of operators that give significant responses to edges (i.e., signal-to-noise ratio above the
thresh- old); in this case the operator with the smallest scale is chosen, as it gives the
best localization of the edge.
Canny proposed a feature synthesis approach. All significant edges
fromtheoperatorwiththesmallestscalearemarkedfirst,andtheedgesof a hypothetical
operator with larger σ are synthesized from them (i.e., a prediction is made of how the
large σ should perform on the evidence gleaned from the smaller σ. Then the
synthesized edge response is compared with the actual edge response for larger σ.
Additional edges are marked only if they have a significantly stronger response than
that predicted from synthetic output.
This procedure may be repeated for a sequence of scales, a cumulative edge map being
built by adding those edges that were not identified at smaller scales.

Figure 5.23a shows the edges of Figure 5.9a detected by a Canny operator with σ =1.0.
Figure5.23b shows theedgedetectorresponsefor σ =2.8 (feature synthesis has not been
applied here).
(a) (b)
Figure5.23:Cannyedgedetectionattwodifferentscales.©CengageLearning 2015.

Canny’s detector represents a complicated but major contribution to edge detection. Its
full implementation is unusual, it being common to find
implementationsthatomitfeaturesynthesis—thatis,juststeps1–5ofAlgorithm 5.4.

2.7. Parametricedgemodels:
Parametric models are based on the idea that the discrete image intensity
functioncanbeconsideredasampledandnoisyapproximationofanunderlying continuous
or piecewise continuous image intensity function.
While this function is not known, it can be estimated from the available discrete image
intensity function and image properties can be determined from this continuous
estimate, possibly with subpixel precision.
It is usually impossible to represent image intensities using a single continuous function
since a single function leads to high-order intensity functions in x and y. Instead,
piecewise continuous function estimates called
facetsareusedtorepresent(aneighborhoodof)eachimagepixel.Suchanimage
representation is called a facet model.
The intensity function in a neighborhood can be estimated using models of different
complexity.
The simplest one is the flat facet model that uses piecewise constants and each pixel
neighborhood is represented by a flat function of constant intensity. The sloped model
uses piecewise linear functions forming a sloped plane fitted to local image intensities.
Quadraticandbi-cubic facetmodelsemploymorecomplexfunctions.
Once the facet model parameters are available for each image pixel, edges
canbedetectedasextremaofthefirstdirectionalderivativeand/orzero-crossings
oftheseconddirectionalderivativeofthelocalcontinuousfacetmodelfunctions.
An example will illustrate: consider a bi-cubic facet model g(i, j) = c1+ c2x + c3y + c4x2 +
c5x y + c6y2 + c7x3 + c8x2 y + c9x y2 + c10y3 ,
(5.59)whoseparametersareestimated from a pixel neighborhood (the co-ordinates of the
central pixel are (0,0)). This may be performed by, e.g., a least-squares method with
SVD; alternatively, coefficients cican be computed directly using a set of ten 5x5 kernels.
Once parameters are available at each pixel, edges may be located as extrema of the first
directional derivative, or zero crossings of the second derivative, of the local facet model
functions.
Benefits:
1) Edgedetectorsbasedonparametricmodelsdescribeedgesmore precisely than
convolution-based edge detectors.
2) Theycarrythepotentialforsubpixeledgelocalization. Limitations:
• Theircomputationalrequirementsaremuch higher.
• PromisingextensionscombinefacetmodelswithCanny’sedgedetection criteria and
relaxation labeling.

2.8. Edgesinmulti-spectralimages:
Onepixelinamulti-spectralimageisdescribedbyann-dimensionalvector, and brightness
values in n spectral bands are the vector components. There are several possibilities for
the detection of edges in multi-spectral images.
Trivially, we might detect edges separately in individual image spectral components using
the ordinary local gradient operators. Individual images of edges can be combined to get
the resulting image, with the value corresponding to edge magnitude and direction
being a selection or combination of the individual edge spectral components.
Alternatively, we may create a multi-spectral edge detector which uses
brightnessinformationfromallnspectralbands;thisapproachisalsoapplicable tomulti-
dimensionalimagesformingthree-orhigher-dimensionaldatavolumes. The neighborhood
used has size 2x n pixels, where the 2x 2 neighborhood is similar to that of the Roberts
gradient, equation (5.39). The coefficients weighting the influence of the component
pixels are similar to the correlation coefficients.Let f(i, j) denote the arithmetic mean of
the brightnesses corresponding to the pixels with the same co-ordinates (i, j) in all n
spectral component images, and frbe the brightness of the rth spectral component. The
edge detector result in pixel (i, j) is given as the minimum of the following expression:

2.9. Localpre-processinginthefrequency domain:

The Fourier transform makes convolution of two images in the frequency
domainveryeasy,anditisnaturaltoconsiderapplyingmanyofthefiltersinthe frequency
domain. Such operations are usually called spatial frequency filtering.

AssumethatfisaninputimageandFisitsFouriertransform.Aconvolution filter h can be

represented by its Fourier transform H; h may be called the unit pulse response of the
filter and H the frequency transfer function, and either of
therepresentationshorHcanbeusedtodescribethefilter.TheFouriertransform of the filter
output after an image f has been convolved with the filter h can be computed in the
frequency domain
G=F. , (5.61)
where .represents an element∗ -by-element multiplication of matrices F and H (not matrix
multiplication).The filtered image g can be obtained by applying the inverse
FouriertransformtoG—equation(3.28).
Some basicexamplesofspatialfilteringare linearlow-pass,high-pass,and
band-passfrequency filters.
1) A low-pass filter is defined by a frequency transfer function H(u, v) with small values at
points located far from the co-ordinate origin in the
frequencydomain(thatis,smalltransfervaluesforhighspatialfrequencies) and large values
 at points close to the origin (large transfer values for low spatialfrequencies)—
seeFigure5.24a.Itpreserveslowspatialfrequencies and suppresses high spatial
frequencies, and has behavior similar to smoothing by standard averaging—it blurs
sharp edges.

(a)(b) (c)
Figure5.24:Frequencyfiltersdisplayedin3D.(a)Low-passfilter.(b)High- pass filter. (c) Band-
pass filter.
2) Ahigh-passfilterisdefinedbysmalltransferfunctionvalueslocatedaround the frequency
co-ordinate system origin, and larger values outside this area— larger transfer
coefficients for higher frequencies (Figure 5.24b).
• Band-passfilters,whichselect frequencies ina certain rangefor
enhancement,areconstructedin asimilarway,andalsofilters with directional response,
etc. (Fig- ure 5.24c).
Themost common imageenhancement problems includenoise suppression, edge
enhancement, and removal of noise which is structured in the frequency spectrum.
Noise represents a high-frequency image component, and it may be suppressed
applying a low-pass filter as shown in Figure 5.25, which demonstrates the principles of
frequency filtering on Fourier image spectra; the
originalimagespectrumismultipliedbythefilterspectrumandalow-frequency image
spectrum results. Unfortunately, all high-frequency phenomena are
suppressed,includinghighfrequenciesthatarenotrelatedtonoise(sharpedges, lines, etc.).
Low-pass filtering results in a blurred image.

(a) (b)
(c)(d)
Figure 5.25: Low-pass frequency-domain filtering—for the original image and its spectrum
see Figure 3.7. (a) Spectrum of a low-pass filtered image, all higher frequencies filtered
out.
(b) Image resulting from the inverse Fourier transform applied to spectrum (a).
(c) Spectrum of a low-pass filtered image, only veryhighfrequenciesfilteredout.
(d) Inverse Fourier transform applied to spectrum (c).

Again, edges represent a high-frequency image phenomenon. Therefore, to enhance them,

low-frequency components of the image spectrum must be suppressed—to achieve this,
a high-frequency filter must be applied.
Toremovenoisewhichisstructuredinthefrequencydomain,thefilterdesign
mustincludeaprioriknowledgeaboutthenoiseproperties.Thisknowledgemay be acquired
either from the image data or from the corrupted image Fourier spectrum, where the
structured noise usually causes notable peaks.
SomeexamplesoffrequencydomainimagefilteringareshowninFigures5.25–
5.28.TheoriginalimagewasshowninFigure3.8anditsfrequencyspectruminFigure
3.7. Figure 5.26 shows results after application of a high-pass filter followed by an inverse
Fourier transform. It can be seen that edges represent high-frequency phenomena in
the image. Results of band-pass filtering can be seen in Figure 5.27. Figure 5.28 gives an
even more powerful example of frequency filtering—removal of periodic noise. The
vertical noise lines in the original image are transformed into
frequencyspectrumpeaksafterthetransform.Toremovethesefrequencies,afilterwas
designed which suppresses the periodic noise in the image, which is visible as white
circular areas.

(a) (b)
(c)(d)
Figure 5.26: High-pass frequency domain filtering. (a) Spectrum of a high- pass filtered
image, only very low frequencies filtered out. (b) Image resulting from the inverse
Fourier transform applied to spectrum (a). (c) Spectrum of a high-pass filtered image,
all lower frequencies filtered out. (d) Inverse Fourier transform applied to spectrum (c).

There are several filters which prove useful for filtering in the frequency domain: two
important representatives of them are the Gaussian and Butterworth filters. Choose an
isotropic filter for simplicity,

,andlet D0beaparameterofthefilter calledthecut-

offfrequency.FortheGaussian,D0coincideswiththedispersionσ.TheFourier
spectrumofalow-passGaussianfilterGlowis
(a) (b)
Figure 5.27: Band-pass frequency domain filtering. (a) Spectrum of a band- passfiltered
image, low and high frequencies filtered out. (b) Image resulting from the inverse
Fourier transform applied to spectrum (a).

TheButterworthfilterisspecifiedtohavemaximallyflatfrequencyresponse
overaspectrumband,andisalsocalleda‘maximallyflatmagnitudefilter’.The frequency
response of the 2D low-pass Butterworth filter Blowof degree n is

TheusualButterworthfilterdegree isn= 2,whichwillbe usedhere.Figure

5.29illustratestheshapeoftheGaussianandButterworthfiltersforD0=3in1D plots.
The high-pass filter is created easily from the low-pass filter. If the Fourier
frequencyspectrumofalow-passfilterisHlow,thehigh-passfiltercanbecreated by just
flipping it vertically, H− high= 1-Hlow.
Anotherusefulpre-processingtechniqueoperatinginthefrequencydomainis
an instance of homomorphic filtering. Homomorphic filtering is used to remove
multiplicative noise.The aim of the particular homomorphic filtering is to
simultaneously increase contrast and normalize image intensity across the image.
Theassumptionisthattheimagefunctionf(x,y)canbefactorizedasaproduct of two
independent multiplicative components in each pixel: illumination i(x, y) and the
reflectance r(x, y) at the point in the observed scene, f (x, y) = i(x, y) r(x, y). These two
components can be separated in some images because the
illuminationcomponenttendstovaryslowlyandthereflectancecomponentvaries more
quickly.The idea of the separation is to apply a logarithmic transform to the input image
z(x, y) = log f (x, y) = log i(x, y) + log r(x, y) . (5.64) If
the image z(x, y) is converted to Fourier space (denoted by capital letters)
thenitsadditivecomponentsremainadditiveduetothelinearityoftheFourier
transform
Z(u, v)=I(u, v)+R(u, v). (5.65)
(a) (b)

(c)
Figure 5.28: Periodic noise removal. (a) Noisy image. (b) Image spectrum
usedforimagereconstruction—notethattheareasoffrequenciescorresponding with
periodic vertical lines are filtered out. (c) Filtered image. © Cengage Learning 2015.
AssumethattheFourierspectrumZ(u,v)isfilteredbythefilterH(u,v)and the spectrum S(u, v) is
the result
S=H . Z=H. I+ H . R. (5.66)
Usually a high-pass filter is used for this purpose; assuming a high-pass Butterworth filter,
it has to be damped in order not to suppress low frequencies
entirelyastheybearneededinformationtoo.TheButterworthfiltermodifiedby damping
coefficient 0.5 is shown in Figure 5.30

Having the filtered spectrum S(u, v), we can return to spatial coordinates
usingtheinverseFouriertransform,s(x,y)=−1S(u,v).Recallthatthelogarithm was first
applied to the input image f (x, y) in equation (5.64). Now the image has to be
transformed by the logarithm inverse function; thisinverse function is the exponential.
The result—the image g(x, y) filtered by the homomorphic filter—is given by g(x, y) =
exp s(x, y) .
Anillustration of the effect of homomorphic filtering is in Figure 5.31, an
imageofapersoninadarktunnelwithstrongilluminationattheentrance.Detail
ofthetunnelsurfaceonthetopandrightsidearenotvisiblebecausethesurface
istoodark.Theresult ofhomomorphicfilteringisinFigure5.31b.Moredetails can be seen in
this image.

(a) (b)
Figure 5.31: Illustration of homomorphic filtering. (a) Original image. (b) Homomorphic
filtering.
2.10. Line detection by local pre-processing operators:
Several other local operations exist which do not belong to the taxonomy
giveninSection5.3,astheyareusedfordifferentpurposessuchaslinefinding, line thinning,
and line filling operators. Another group of operators finds ‘interest points’or
‘locations of interest’in the image.
It is interesting to seek features richer than edges which can be reliably detected in the
image and which can outperform simple edge detectors in some classes of applications.
Line detectors and corner detectors are some such. Line
detectorsareusedtodetectlinearobjectssuchasdimensionlinesinengineering drawings or
railways or roads in satellite images. Corner detectors and other interest point-like
detectors are used mainly to register two or more images one to the other (e.g, in stereo
vision, motion analysis, panorama stitching, object recognition from images) or to index
the image or dominant objects in it to an image database.
Line finding operators aim to find very thin curves in the image; it is
assumedthatcurvesdonotbendsharply.Suchcurvesandstraightlinesarecalled
linesforthepurposeofdescribingthistechnique.Ifacrosssectionperpendicular
indirectiontothetangentofalineisexamined,wegetaroofprofile(seeFigure 5.17) when
examining edges. We assume that the width of the lines is approximately one or two
pixels.
Thepresenceofalinemaybedetectedbylocalconvolutionoftheimagewith con-
volutionkernelswhichserveaslinepatterns.Thesimplestcollectionoffour such patterns of
size 3 x 3 is able to detect lines rotated modulo the angle 45o. Three of four such
convolution kernels are

Asimilarprinciplecanbeappliedtobiggermasks.Thecaseof5x5masksis common. Such line

detectors sometimes produce more lines than needed, and other non-linear constraints
may be added to reduce this number. More sophisticatedapproachesdeter-
minelinesinimagesasridgesandravinesusing the facet model. Line detection is frequently
used in remote sensing and in document processing;
Local information about edges is the basis of a class of image segmentation techniques.
Edges which are likely to belong to object boundaries are usually found by simple
thresholding of the edge magnitude—such edge thresholding does not provide ideal
contiguous boundaries that are one pixel wide. Sophisticated segmentation techniques
that are dealt with in the next chapter serve this purpose. Here, much simpler edge
thinning and filling methods are described. These techniques are based on knowledge of
small local neighborhoods and are very similar to other local pre-processing techniques.
Thresholded edges are usually wider than one pixel, and line thinning
techniquesmaygiveabetterresult.Onelinethinningmethodusesknowledgeabout edge
orientation and in this case edges are thinned before thresholding. Edge
magnitudesanddirectionsprovidedbysomegradientoperatorareusedasinput,and
the edge magnitudes of two neighboring pixels perpendicular to the edge direction are
examined for each pixel in the image. If at least one of these pixels has edge magnitude
higher than the edge magnitude of the examined pixel, then the edge magnitude of the
examined pixel is assigned a zero value.
Therearemanyotherlinethinningmethods.Inmostcasesthebestresultsare achieved using
mathematical morphology methods.

2.11. Imagerestoration:
Pre-processing methods that aim to suppress degradation using knowledge about its
nature are called image restoration. Most image restoration methods are based on
convolution applied globally to the whole image. There is a wide literature on
restoration and only the basic principles and some simple degradations are considered
here.
Image degradation can have many causes: defects of optical lenses,
nonlinearityoftheelectro-opticalsensor,graininessofthefilmmaterial,relative motion
between an object and camera, wrong focus, atmospheric turbulence in remote sensing
or astronomy, scanning of photographs, etc. The objective of image restoration is to
reconstruct the original image from its degraded version.
Image restoration techniques can be classified as deterministic or stochastic.
Deterministic methods are applicable to images with little noise and a known
degradation function. The original image is obtained by applying the function inverse to
the degraded one. Stochastic techniques try to find the best restoration
accordingtoaparticularstatisticalcriterion,e.g.,aleast-squaresmethod.Thereare
threetypicaldegradationswithasimplefunction:relativeconstantspeedmovement of the
object with respect to the camera, wrong lens focus, and atmospheric turbulence.
Inmostpracticalcases,thereisinsufficientknowledgeaboutthedegradation, and it must be
estimated and modeled. This may be done on an a priori or a posteriori basis:
2.11.1. Apriori knowledge about degradation is either known in advance or
canbeobtainedbeforerestoration.Forexample,ifitisknownthatthe image was degraded
by relative motion of an object with respect to the sensor, then the modeling determines
only the speed and direction of the motion. Alternatively, wemay seek to to
estimateparameters ofadevicesuch as a TV camera or digitizer, whose degradation
remains unchanged over a
periodoftimeandcanbemodeledbystudyingaknownsampleimageand its degraded
version.
2.11.2. A posteriori knowledge is that obtained by analyzing the
degradedimage.Atypical example is to find some interest points in the image
(e.g.,corners,straightlines)andguesshowtheylookedbefore
degradation.Another possibility is to use spectral characteristics of the regions in the
image that are relatively homogeneous.
Adegradedimagegcanarisefromtheoriginalimagefbyaprocesswhichcanbe expressed as
where s is some non-linear function and ν describes the noise. This is often simplified by
neglecting the non-linearity and assuming that the function h is invariant with respect
to position in the image, giving
g(i,j) =(f h)(i,j) +ν(i, j) . (5.75)
If the noise is not significant in this equation, then restoration equates to inverse
convolution (also called deconvolution). If noise is not negligible, then the inverse
convolution is solved as an overdetermined system of linear equations. Methods based
on minimization of least square error such asWiener filtering(off-
line)orKalmanfiltering(recursive,on-line;seeSection16.6.1)are examples [Bates and
McDonnell, 1986].

2.11.1. Degradationsthatareeasyto restore

IntheFourierdomain,wecanexpressequation(5.75) as
G=HF. (5.76)
Therefore,overlookingimagenoiseν,knowledgeofthedegradationfunction fully facilitates
image restoration by inverse convolution (Section 5.4.2).
Relativemotionofcameraand object
Relativemotionofacamerawithamechanicalshutterandthephotographed
objectduringtheshutteropentimeTcausessmoothingoftheobjectintheimage. Suppose V is
the constant speed in the direction of the x axis; the Fourier transform H(u, v) of the
degradation caused in time T.

Wronglensfocus

Image smoothing caused by imperfect focus of a thin lens can be described by the function

where J1is the Bessel function of the first order, r2 = u2 + v2, and a is the displacement— the
model is not space invariant.
Atmosphericturbulence
Atmospheric turbulence is degradation that needs to be restored in remote
sensingandastronomy.Itiscausedbytemperaturenon-homogeneityinthe
atmospherethatdeviates passinglightrays.Onemathematicalmodel[Hufnagel and Stanley,
1964]
is

H(u,v)= e−c(u2+v2)5/6 , (5.79)

wherecisaconstantthatdependsonthetypeofturbulencewhichisusually found
experimentally. The exponent 5/6 is sometimes replaced by 1.
2.11.2. Inverse filtering
Inverse filtering assumes that degradation was caused by a linear function h(i,j)
(cf.equation 5.75) and considerstheadditivenoise ν as anothersourceof degradation. It is
further assumed that ν is independent of the signal. After applying the Fourier
transform to equation (5.75), we get
G(u,v)=F(u, v) H(u,v)+N(u,v). (5.80)
Thedegradation can beeliminatedusing therestoration filterwith atransfer function that is
inverseto the degradation h.We derivethe original image F (its Fourier transform to be
exact) from its degraded version G (equation 5.80), as
F (u, v) =G(u, v)H−1(u, v)−N (u, v) H−1(u, v). (5.81)

Thisshowsthatinversefilteringworkswellforimagesthatarenotcorrupted by noise [not

considering possible computational problems if H(u, v) gets close to zero at some
location of the u, v space—fortunately, such locations can be
neglectedwithoutperceivableeffectontherestorationresult].However,ifnoise
ispresent,twoproblemsarise.First,thenoiseinfluencemaybecomesignificant for
frequencies where H(u, v) has small magnitude. This situation usually corresponds to
high frequencies u, v. In reality, H(u, v) usually decreases in magnitude much more
rapidly than N (u, v) and thus the noise effect may dominate the entire restoration
result. Limiting the restoration to a small neighborhoodofthe u,v
origininwhichH(u,v)issufficiently largeovercomes this problem, and the results are
usually quite acceptable. Secondly, we usually
donothaveenoughinformationaboutthenoisetodetermineN(u,v)sufficiently.

2.11.3. Wienerfiltering
Wiener (least mean square) filtering [Wiener, 1942; Gonzalez and Woods, 1992; Castle-
man, 1996] attempts to take account of noise properties by incorporatingaprioriknow-
ledgeintheimagerestorationformula.Restoration
bytheWienerfiltergivesanestimatefˆoftheoriginaluncorruptedimagefwith minimal
mean square error
! "

(5.82)
wheredenotesthemeanoperator.𝖲Ifnoconstraintsareappliedtothesolution of
equation (5.82), then an optimal estimate fˆ is the conditional mean value of the
ideal image f under the condition g. This approach is complicated from the
computational point of view. Moreover, the conditional probability density
between the optimal image f and the corrupted image g is not usually known.
The optimal estimate is in general a non- linear function of the image g.
ˆ
Minimization of equation (5.82) is easy if the estimate f is a linear combination of
the
values in image g; the estimatefˆ is then close (but not necessarily equal) to the
theoretical optimum. The estimate is equal to the theoreticalopti
DenotetheFouriertransformoftheWienerfilter byHW.Then,theestimate Fˆ of the
Fourier transform F of the original image f can be obtained as

HWisnotderivedhere,butmaybefoundelsewhere[GonzalezandWoods,1992]as

where H is the transform function of the degradation,denotes complex conjugate,

Sννis the spectral density of the noise, and Sffis the spectral density of the
undegradedimage.
If Wiener filtering is used, the nature of degradation H and statistical parameters of
the noise need to be known. Wiener filtering theory solves the problem of
optimal a posteriori linear mean square estimates—all statistics (for example,
power spectrum)
should be available in advance. Note the term Sff(u, v) in equation (5.84), whichrep-
resentsthespectrumoftheundegradedimage,whichmaybedifficult to obtain with
no foreknowledge of the undegraded image.
RestorationisillustratedinFigure5.36whereanimagethatwasdegradedby 5 pixels
motion in the direction of the x axis: Figure 5.36b shows the result of restoration
by Wiener filtering.

(a) (b)
Figure5.36:RestorationofmotionblurusingWienerfiltering.CourtesyofP.Kohout,
Criminalistic Institute, Prague.