Page 74 - Classification Parameter Estimation & State Estimation An Engg Approach Using MATLAB

PERFORMANCE OF ESTIMATORS 63

3.2.1 Bias and covariance

The error e is composed of two parts. One part is the one that does not
change value if we repeat the experiment over and over again. It is the
expectation of the error, called the bias. The other part is the random
part and is due to sensor noise and other random phenomena in the
sensory system. Hence, we have:

error ¼ bias þ random part

If x is a scalar, the variance of an estimator is the variance of e. As such
the variance quantifies the magnitude of the random part. If x is a vector,
each element of e has its own variance. These variances are captured in
the covariance matrix of e, which provides an economical and also a
more complete way to represent the magnitude of the random error.
The application of the expectation and variance operators to e needs
some discussion. Two cases must be distinguished. If x is regarded as a
non-random, unknown parameter, then x is not associated with any
probability density. The only randomness that enters the equations is
due to the measurements z with density p(zjx). However, if x is regarded
as random, it does have a probability density. We have two sources of
randomness then, x and z.
We start with the first case which applies to, for instance, the max-
imum likelihood estimator. Here, the bias b(x) is given by:

def
x
bðxÞ¼E½^ x xjxŠ
ð3:35Þ
Z
x
¼ ð^ xðzÞ xÞpðzjxÞdz
The integral extends over the full space of z. In general, the bias depends
on x. The bias of an estimator can be small or even zero in one area of x,
whereas in another area the bias of that same estimator might be large.
In the second case, both x and z are random. Therefore, we define an
overall bias b by taking the expectation operator now with respect to
both x and z:

def
x
b¼E½^ x xŠ
ð3:36Þ
ZZ
x
¼ ð^ xðzÞ xÞpðx; zÞdzdx
The integrals extend over the full space of x and z.