ADAPTIVE FILTERS

Ali H. Sayed

Electrical Engineering Department

University of California, Los Angeles

A. H. Sayed, Adaptive Filters, Wiley, NJ, 2008.

ADDITIONAL PROBLEMS

This file contains additional problems that instructors may find useful. The file is updated on
a regular basis. Solutions are available to instructors only. Please contact the author at
[email protected].
ii ADDITIONAL PROBLEMS
Last updated February 2010

1. (Chapters 1, 3, 5) Consider two real-valued scalar random variables x1 and x2 . The

random variable x1 assumes the value +1 with probability p and the value −1 with
probability 1 − p. The random variable x2 is distributed as follows:
½
+2 with probability q
if x1 = +1 then x2 =
−2 with probability 1 − q
½
+3 with probability r
if x1 = −1 then x2 =
−3 with probability 1 − r

Consider further the variables

y 1 = x1 + v 1 and y 2 = x1 + x2 + v 2

where {v 1 , v 2 } are independent zero-mean Gaussian random variables with unit

variance. The variables {v 1 , v 2 } are independent of x1 and x2 .

(a) Express the pdfs of the individual random variables x1 and x2 in terms of delta
functions.
(b) Find the joint pdf of (x1 , x2 ).
(c) Find the joint pdf of (y 1 , y 2 ).
(d) Find the joint pdf of (x1 , x2 , y 1 , y 2 ).
(e) Find the conditional pdf of (x1 , x2 ) given (y 1 , y 2 ).
(f) Find the minimum mean-square error estimator of x2 given {y 1 , y 2 }.
(g) Find the minimum mean-square error estimator of x2 given {y 1 , y 2 , x1 }.
(h) Find the linear least-mean-squares error estimator of x2 given {y 1 , y 2 , x1 }.

2. (Chapters 15, 16) Consider a NLMS recursion of the form

µu∗i
wi = wi−1 + [d(i) − ui wi−1 ]
kui k2

with 1 × M regression vectors ui and step-size µ. Each entry of ui has the form
rej θ , where θ is uniformly distributed over [0, 2π] and r > 0. In other words, the
entries of ui lie on a circle of radius r. Assume the data d(i) satisfy the stationary
data model of Section 6.2.

(a) Find an exact expression for the EMSE of NLMS under such conditions.
(b) Does the value of r have an influence on the EMSE? Is there an optimal choice
for r?
(c) The entries of the regression vectors are further assumed to be independent of
each other. Find an exact condition on the step-size µ to ensure mean-square
convergence.
 (d) Which algorithm will have the lower MSE in steady-state for the same step- iii
size: LMS or NLMS?
(e) Evaluate the number of iterations that are needed for LMS to be within 5% of
its EMSE? What about NLMS? Which algorithm converges faster? Assume
the same step-size for both algorithms.

3. (Chapters 35, 40, 41) Consider a least-squares problem of the form

 
i
X
min τ λi+1 kwM k2 + λi−j |d(j) − uM,j wM |2 
wM
j=0

where τ > 0 is a regularization parameter, 0 ¿ λ ≤ 1 is a forgetting factor, and

uM,j is a 1×M regression vector. We denote the solution vector by wM,i . Introduce
the data matrix  
uM,0
 uM,1 
 
HM,i =  .  ((i + 1) × M )
 .. 
uM,i
and partition HM +1,i as follows:
£ ¤ £ ¤
HM +1,i = x0,i H̄M,i = HM,i xM,i

where {x0,i , xM,i } denote the leading and trailing columns of HM +1,i . It is assumed
that the regression data satisfy the following structural relation:
· ¸
0
H̄M,i =
HM,i−1 Φ−1
M

where ΦM is an M × M invertible matrix of the form

· ¸
FM −1
ΦM =
µ

where FM −1 is a unitary matrix and µ is a scalar.

(a) Derive an a-posteriori-based lattice filter for order-updating the least-squares

solution.
(b) Derive an array-based lattice filter for order-updating the same least-squares
solution.
(c) (Optional) Can you think of a situation where such structural relations arise?

4. (Chapters 29, 30) Consider a least-squares problem of the form

 
N
X
min δkwk2 + λN −j |d(j) − uj w|2 
w
j=0
iv where δ > 0 is a regularization parameter, uj is a 1 × M regression vector, and
0 ¿ λ ≤ 1 is a forgetting factor defined as follows:
½
λe for j even
λ=
λo for j odd

Let wN denote the solution to the above least-squares problem. Can you derive a
recursive least-squares solution that updates wN to wN +1 ?

5. (Chapters 2, 4) Consider noisy observations y(i) = x + v(i), where x and v(i) are
independent random variables, v(i) is a white random process with zero mean and
distributed as follows:

v(i) is Gaussian with variance σv2 with probability q

v(i) is uniformly distributed over [−a, a] with probability 1 − q

Moreover, x assumes the values {1+j, 1−j, −1+j, −1−j} with equal probability.
The value of x is the same for all measurements {y(i)}.

(a) Find an expression for the optimal least-mean-squares estimate of x given a

collection of N observations {y(0), y(1), . . . , y(N − 1)}.

(b) Find the linear least-mean-squares estimate of x given the combined observa-
tions {y(0), y(1), . . . , y(N − 1)} and {y 2 (0), y 2 (1), . . . , y 2 (N − 1)}.

6. (Chapters 3, 4) Let x be a zero-mean random variable with an M × M positive-

definite covariance matrix Rx . Let x̂y1 denote the linear least-mean-squares estima-
tor of x given a zero-mean observation y 1 with covariance matrix Ry1 . Likewise,
let x̂y2 denote the linear least-mean-squares estimator of x given another zero-mean
observation y 2 with covariance matrix Ry2 . Let Ry1 ,y2 = E y 1 y ∗2 . We want to de-
termine another estimator for x by combining x̂y1 and x̂y2 in a convex manner as
follows:
x̂ = λx̂y1 + (1 − λ)x̂y2
where λ is a real scalar lying inside the interval 0 ≤ λ ≤ 1.
(a) Determine the value of λ that results in an estimator x̂ with the smallest mean-
square error.
(b) If λ is allowed to be any arbitrary real scalar (not necessarily limited to the
range [0, 1]), how much smaller can the mean-square-error be?

7. (Chapters 8, 10) Let d denote a scalar zero-mean random variable with variance
σd2 , and let u denote a 1 × M zero-mean random vector with covariance matrix
Ru = E u∗ u > 0. Consider the optimization problem
M
X
min E |d − uw|2 subject to c(k)w(k) = 1
w
k=1

where the {w(k)} denote the individual entries of w and the {c(k)} are scaling
coefficients.
 v
(a) Derive a stochastic-gradient algorithm for approximating the optimal solution
wo in terms of realizations {d(i), ui } for {d, u}, and starting from an initial
condition w−1 that satisfies the constraint.
(b) Derive an approximate expression for the EMSE of the filter for sufficiently
small-step-sizes.
(c) Derive an optimal choice for the coefficients {c(k)} in order to result in the
smallest EMSE.
(d) (Bonus) Can you repeat parts (a)-(c) when the {c(k)} are required to be non-
negative scalars?

8. (Chapters 15, 16, 23) Consider the following constrained LMS recursion
· ¸
cc∗
wi = wi−1 + µ I − u∗ (d(i) − ui wi−1 ), c∗ w−1 = 1
kck2 i

which results from considering an instantaneous approximation for the following

linear least-mean-squares estimation problem:
M
X
min E |d − uw|2 subject to c(k)w(k) = 1
w
k=1

where the {w(k)} denote the individual entries of w and the {c(k)} are the scalar
entries of the column vector c. Moreover, d denotes a scalar zero-mean random
variable with variance σd2 , and u denotes a 1 × M zero-mean random vector with
covariance matrix Ru = E u∗ u > 0. Assume all data are circular Gaussian.

(a) Perform a transient mean-square-error analysis of the adaptive filter and pro-
vide conditions on the step-size µ in order to ensure that the filter is mean-
square stable. Specify clearly the conditions on the data that you are assuming
for your analysis.
(b) Derive expressions for the EMSE and the MSD of the filter.
(c) Derive an expression for the learning curve of the filter.

9. (Chapters 29, 32) Let w

b denote the solution to the following regularized least-
squares problem

min [w∗ Πw + (y − Hw)∗ W (y − Hw)]

w

where W > 0 and Π > 0. Let yb = H wb denote the resulting estimate of y and let ξ
denote the corresponding minimum cost. Now consider the extended problem

( " # " # !∗ " # " # !)

y ha H hb y ha H hb
min wz∗ Πz wz + − wz Wz − wz
wz d αa u αb d αa u αb

where a and b are positive scalars, ha and hb are column vectors, αa and αb are
scalars, d is a scalar, u is a row vector, and
 
a · ¸
  W
Πz = Π , Wz =
1
b
vi Let · ¸
ha H hb
ybz = w
bz
αa u αb
and let ξz denote the corresponding minimum cost of the extended problem.
(a) Relate {w
bz , ybz , ξz } to {w,
b yb, ξ}.
(b) Can you motivate and derive an array algorithm to update the solution from w
b
to w
bz ?

10. (Chapter 30) Let wi denote the solution to the following regularized least-squares
problem: £ ¤
min λi+1 w∗ Πw + kyi − Hi wk2
w

where λ is a real scalar such that 0 < λ ≤ 1, Π is a positive-definite diagonal matrix,

and where yi and Hi are related to yi−1 and Hi−1 as follows:
· ¸ · ¸
yi−1 Hi−1
yi = Hi =
d(i) ui

with d(i) denoting a scalar and ui denoting a row vector. We also define the matrix
£ ¤−1
Pi = λi+1 Π + Hi∗ Hi

a) Assume that at time i, it holds that Hi∗ Hi > 0. Let wu,i denote the solution to
the following un-regularized least-squares problem:

min kyi − Hi wk2

w

and let Pu,i = [Hi∗ Hi ]−1 . Provide a recursive algorithm to compute the un-
regularized quantities {wu,i , Pu,i } from the regularized quantities {wi , Pi }.
b) Derive a recursive algorithm to update the regularized solution, i.e., to compute
{wi , Pi } from {wi−1 , Pi−1 }.

In both parts (a) and (b), your algorithm should use a series of rank-1 updates, and
direct matrix inversions are not allowed.