PROPERTIES OF ESTIMATORS

1. Unbiased: E (W) = (centered around true

value)
2. Consistent: W for large number of
observations
3. Efficient: W1 is better than W2 if
4. Minimum Variance: efficient AND
achieve CRLB
5. Sufficient statistic for
2 2
) 2 ( ) 1 ( u u < W E W E

u
Let be an estimator for , we say
that is unbiased for if E( )= .
If E( )= +b() with b()0, then
is a biased estimator for with
bias b().

u

u

u
If one writes of estimation, then an
accurate estimator would be the one resulting in
small estimation errors, so that estimated values will
be near the true value.
error + =u u

If an estimator t
n
estimates u then difference
between them (t
n
-u ) is called the estimation
error. Bias of the estimator is defined as the
expectation value of this difference:
B
u
=E(t
n
-)=E(t
n
)- u
If the bias is equal to zero then the estimation
is called unbiased. For example sample mean
is an unbiased estimator:
0 ) (
1
) (
1
)
1
( ) (
1 1 1
= = = =

= = =

n
i
n
i
i
n
i
i
x E
n
x E
n
x
n
E x E
Here we used the fact that expectation and
summation can change order (Remember that
expectation is integration for continuous random
variables and summation for discrete random
variables.) and the expectation of each sample
point is equal to the population mean.
Knowledge of population distribution was not
necessary for derivation of unbiasedness of the
sample mean. This fact is true for the samples
taken from population with any distribution for
which the first moment exists..
A biased estimator can be an unbiased
estimator when n is large.
If , then the estimator is
asymptotically unbiased.
u

lim ( ) 0
n
b u

=
Suppose that X
1
, X
2
,, X
n
constitute a random
sample from a distribution with mean
1
and
standard deviation
1
, and that Y
1
, Y
2
,, Y
n
(independent of X
i
s) constitute a random sample
from a distribution with mean
2
and standard
deviation
2
.
Use the rules of expected value to show that X Y
is an unbiased estimator of
1
-
2
.
Solution:
2 1
] [ ] [ ] [
=
= Y E X E Y X E
We have shown that when sampling from
, one finds that the maximum likelihood estimators of and
are
Recalling that the distribution of
Thus, we know that the distribution of
Hence,
2
1 2
( , ) N u u o = =

2
o
2
2
1 2
( 1)

and
n S
X
n
u u o

= = = =
2
is ( , ), we see that ( ) ; X N E X
n
o
=
is unbiased estimator of . X
2 2 2 2
1
2
1
( 1) 1
is where ( )
1
n
n i
i
n
S S X X
n
_
o

=

2 2 2
2 2
2
( 1)
( ) ( 1)
1 1
n S
E S E n
n n
o o
o
o
(
= = =
(


Therefore, S
2
is an unbiased estimator of o
2
Consequently, since
Therefore,
2 2 2
2
( 1) ( 1) ( 1)

( ) [ ] [ ]
n n n
E E S E S
n n n
u o

= = =
2
2 2

is a biased estimator of u u o =
Expectation value of the square of the differences
between estimator and the expectation of the
estimator is called its variance:
Exercise: What is the variance of the sample mean.
As we noted if estimator for u is t
n
then difference
between them is error of the estimation. Expectation
value of this error is bias. Expectation value of square
of this error is called mean square error (m.s.e.):
2
) ( u
u
=
n
t E M
2
)) ( (
n n
t E t E V =
u
It can be expressed by the bias and the
variance of the estimator:
M.S.E is equal to square of the estimators bias plus
variance of the estimator. If the bias is 0 then MSE is
equal to the variance.
In estimation it is usually trade of between
unbiasedness and minimum variance. In ideal world
we would like to have minimum variance unbiased
estimator. It is not always possible.
) ( ) (
) ) ( ( )) ( ( ) ) ( ) ( ( ) ( ) (
2
2 2 2 2
n n
n n n n n n n n
t B t V
t E t E t E t E t E t E t E t M
u u
u
u u u
+
= + = + = =
UNBIASED!!!!
Let S
2
be a sample variance for a random sample
from and . Is this
estimator biased or unbiased estimator? Note: mean
of chi-square is it df.
Solution:
since E(S
2
)o
2
; therefore S
2
is biased estimator and the
biased value is - o
2
/n.
1 2
, ,...,
n
X X X
2
( , ) N o
2
2
( )
i
x x
s
n

2
2
2 2
1 1
2
2
2 2
2
2
2
2
2
2
2
( )
( 1) ~ ; ~
( )
( ) [ ]
( )
[ ]
( 1)
i
n n
i
i
x x
s
n
n
x x
E s E
n
x x
E
n
n
n
n
_ _
o
o
o
o
o
o
o
o

=
=
=

If an estimate is such that its biased

converged to zero as n, then we say that it
is asymptotically unbiased.
S
2
is asymptotically unbiased when n
2
2 2
2 2
( )
lim ( ) .
n
E s
n
E s
o
o
o

=
=
Solution:
2
2
( )
If , does this estimator is unbiased
1
estimator of ?
i
x x
w
n
o

2
2
2
2
2
2
2
2
( )
( ) [ ]
( 1)
( )
[ ]
1
( 1)
1
i
i
x x
E w E
n
x x
E
n
n
n
o
o
o
o
o
o

w is unbiased estimator for

o
2
THEOREM
Let be an unbiased estimator for u
based on a random sample of size n,
from function , assume we can differentiate
inside the integral whenever needed, then CRLB for
the Var(y) is given by
1 2
( , ,..., )
n
y U x x x =
1 2
, ,...,
n
x x x
( ; ) f x u
2
2
2
2
2
1
ln ( , )
1
ln ( , )
y
f x
nE
f x
n E
o
u
u
u
u
>

c (
`
(
c


)
=

( c
`
(
c


)
Cramer Rao
Lower bound
Corrollary
Suppose y is nor unbiased; say E(y)=u+b(u),
then CRLB for y is:
| |
2
2
2
2
2
1 '( )
ln ( , )
y
b
f x
n E
u
o
u
u
+
>

(
c
`
(
c


)
Get the CRLB and find the variance for this
estimator
1
, 0
( ; ) ; ~ ( 1, )
0 ,
x
e x
f x y gamma
elsewhere
u
o
u o | u
u

< <
= = =

Solution:
2
2
2 2 3
2
2 2 3
2 2
2
ln ( ; ) ln
ln ( ; ) 1
ln ( ; ) 1 2
ln ( ; ) 1 2 ( )
[ ]
1 2
1
x
f x
f x x
f x x
f x E x
E E
u u
u
u
u u u
u
u u u
u
u u u
u u
u
=
c
= +
c
c
= +
c
( c
= +
(
c

= +
=
2
2
2
2
1
1
CRLB:
y
y
n
n
o
u
u
o
>
(
(

>
2 2 2
1
( ) 1.
( ) 1.
; ( )

E X
V X
x
M x E X
n
x
o| u u
o| u u
u
u
= = =
= = =
= = =
=

Therefore, variance for equal to CRLB.

If an unbiased estimator achive the CRLB;
we say that is efficient for estimating u and
is called uniformly minimum- variance (or
best) unbiased estimator
2
2
2
2 2
2 2

( ) ( ) ( )
1
( )
1
( )
1 1
i
i
x
V V x V
n
V x
n
V x
n
n
n n n
u
u
u u
= =
=
=
= = =

u
If we consider all possible unbiased estimator of
some parameter u , the one with the smallest
variance is called the most efficient estimator of u.
1

u
u
2

u
3

u
Sampling distribution of different estimators of u
Unbiased
estimators
is the best
estimator of u
1

u
Let be an estimator that is unbiased for u.
The relative efficiency for is defined to be:

( ) 1

( )
CRLB
eff
V
u
u
= s
If and are biased estimators of parameter of
a given population, the measure of efficiency of
relative to is denoted by the ratio
where
1

u
2

u
2 2

( ) [( ) ] ( ) ( ) MSE E Var b u u u u u = = +
1

u
2

u
2
1 2
1

( )

( , )

( )
MSE
eff
MSE
u
u u
u
=
Mean square error
CAREFUL HERE
What it means Conclusion
2
1 2
1

( )

( , )

( )
MSE
eff
MSE
u
u u
u
=
1 2

( , ) 1 eff u u =
1 2

( , ) 1 eff u u >
1 2

( , ) 1 eff u u <
1 2

( ) ( ) MSE MSE u u =
1 2

( ) ( ) MSE MSE u u <
1 2

( ) ( ) MSE MSE u u >
1

u
2

u
1

u
2

u
1

u
2

u
as efficient as
more efficient
than
less efficient
than
If and are unbiased estimators of parameter
of a given population, and , then we say
that is relatively more efficient than .
The measure of efficiency of relative to is
denoted by the ratio
1

u
2

u
1 2

( ) ( ) Var Var u u <
1

u
2

u
1

u
2

u
2
1 2
1

( )

( , )

( )
Var
eff
Var
u
u u
u
=
CAREFUL HERE
What it means Conclusion
2
1 2
1

( )

( , )

( )
Var
eff
Var
u
u u
u
=
1 2

( , ) 1 eff u u =
1 2

( , ) 1 eff u u >
1 2

( , ) 1 eff u u <
1 2

( ) ( ) Var Var u u =
1 2

( ) ( ) Var Var u u <
1 2

( ) ( ) Var Var u u >
1

u
2

u
1

u
2

u
1

u
2

u
as efficient as
more efficient
than
less efficient
than
Suppose that X
1
and X
2
are random samples
from a normal distribution with mean and
standard deviation . Two estimators for
mean are as below:
Check whether the two estimators and
are unbiased.
Determine the measure of efficiency of
relative to .
1 2
1
2
3
X X

+
=
1 2
2
2
X X

+
=
1

Check whether the two estimators and are

unbiased.
1 2
1
1 2
2
( )
3
1 2
( ) ( )
3 3
1 2
3 3
X X
E E
E X E X

+
| |
=
|
\ .
= +
= +
=
1 2
2
1 2
( )
2
1 1
( ) ( )
2 2
1 1
2 2
X X
E E
E X E X

+
| |
=
|
\ .
= +
= +
=
1

Since both and ,then the two

estimators and are unbiased.
1
( ) E =
2
( ) E =
1

Determine the measure of efficiency of relative to

1 2
1
1 2
2 2
2
2
( )
3
1 4
( ) ( )
9 9
1 4
9 9
5
9
X X
Var Var
Var X Var X

o o
o
+
| |
=
|
\ .
= +
= +
=
1 2
2
1 2
2 2
2
( )
2
1 1
( ) ( )
4 4
1 1
4 4
1
2
X X
Var Var
Var X Var X

o o
o
+
| |
=
|
\ .
= +
= +
=
1

Since both and ,then the two

estimators and are unbiased.
1
( ) E =
2
( ) E =
1

So, the measure of efficiency of relative to

( )
( )
2
1 2
1
2
2
( , )
2
5
9
9
0.9 1
10
Var
eff
Var

o
o
=
=
= = <
1

Since that is ,then the

estimator is 0.9 times more efficient than .
1

1 2

( , ) 1 eff u u <
1 2

( ) ( ) Var Var u u >
Suppose that X
1
, X
2
and X
3
are
random samples from a normal
distribution with mean and standard
deviation . Determine the efficiency
measure of relative to
1 2 3
2
4
X X X + +
1 2 3
3
X X X + +
An estimator is said to be consistent for

if .
P
o u u
An unbiased estimator is said to be
consistent if the difference between the
estimator and the parameter grows smaller
as the sample size grows larger.
E.g. is a consistent estimator of
because: V(X) is
That is, as n grows larger, the variance of
grows smaller.
X
X
Let be a random sample from
function f(x;u), ueO. Let y
1
=U( ) be a
statistic with pdf g(y
1
,u). We say that y
1
is a
sufficient statistic for the family u
if and only if
Where for every fix value of the
func H( ) does not depend on u, thus
the conditional pdf of given
is free of u
1 2
, ,...,
n
x x x
1 2
, ,...,
n
x x x
{ }
( , ),
i
F f x
u
u u = eO
( )
1 2
1 2
1 1
1 2
, ( , )... ( , )
joint pdf for , ,...,
( , ) ( , )
( , ,..., )
n
n
n
f x f x f x
x x x
g y g y
H x x x
u u u
u u
=
=
y
1
=U( )
1 2
, ,...,
n
x x x
1 2
, ,...,
n
x x x
1 2
, ,...,
n
x x x
1 1
Y y =
Let say is a r.s from
1 2
, ,...,
n
x x x
1
1
, 0,1,...
( ; )
!
0 ,
is is a sufficient for ?
x
i
e
x
f x
x
elsewhere
y x
y
u
u
u
u

i. find joint pdf for

ii.
Conc: the given func. Is a sufficient statistic.
1 2
, ,...,
n
x x x
1 2
1 1
( , ) ( , )... ( , ) ( , )
!
i
x
n n
n i
i i
e
f x f x f x f x
x
u
u
u u u u

= =
= =
[ [
1
! ( )!
!
!
( )! ( )!
i
i
i i
x
n
x
n
i i
i
x x
n n
i
i i
e
e
x x
x
n x
n e n e
x x
u
u
u u
u
u
u u

= =

[
[
[

Free of u
Note:
There is another way to find sufficient statistic by
Neymans Factorization Theorem.
Theorem:
Let be a random sample from function
f(x;u), ueO. Let y
1
=U( ) be a statistic of
g(y
1
,u) that y
1
is a sufficient statistic for family u
if and only if there is exist a
function k
1
(.) if k
2
(.)
where for each fix value of y
1
=U( ) the
function k
2
( ) does not depend on u in
form or in domain.
1 2
, ,...,
n
x x x
1 2
, ,...,
n
x x x
{ }
( , ),
i
F f x
u
u u = eO
1 1 1 2 2 1 2
1
( ; ) ( ( , ,..., )) ( , ,..., )
n
n n
i
f x k U x x x k x x x u
=
=
[
1 2
, ,...,
n
x x x
1 2
, ,...,
n
x x x
Is just a statistic
Let is a r.s from
Solution:
1 2
, ,...,
n
x x x
1
2
(2 )
[ (1 )] , 0 1
[ ( )] ( ; )
0 ,
Find sufficient statistic for
x x x
f x
elsewhere
u
u
u u
u

< <

I =