Page 18 - Classification Parameter Estimation & State Estimation An Engg Approach Using MATLAB

THE SCOPE OF THE BOOK 7

Since the nature of the questions raised in the three subjects is similar, the
analysis of all three cases can be done using the same framework. This allows
an economical treatment of the subjects. The framework that will be used is
a probabilistic one. In all three cases, the strategy will be to formulate the
posterior knowledge in terms of a conditional probability (density) function:

Pðquantities of interestjmeasurements availableÞ

This so-called posterior probability combines the prior knowledge with
the empirical knowledge by using Bayes’ theorem for conditional prob-
abilities. As discussed above, the framework is generic for all three cases.
Of course, the elaboration of this principle for the three cases leads to
different solutions, because the natures of the ‘quantities of interest’
differ.
The second similarity between the topics is their reliance on models.
It is assumed that the constitution of the object/physical process/event
(including the sensory system) can be captured by a mathematical model.
Unfortunately, the physical structures responsible for generating the
objects/process/events are often unknown, or at least partly unknown. Con-
sequently, the model is also, at least partly, unknown. Sometimes, some
functional form of the model is assumed, but the free parameters still
have to be determined. In any case, empirical data is needed in order to
establish the model, to tune the classifier/estimator-under-development,
and also to evaluate the design. Obviously, the training/evaluation data
should be obtained from the process we are interested in.
In fact, all three subjects share the same key issue related to modelling,
namely the selection of the appropriate generalization level. The empirical
data is only an example of a set of possible measurements. If too much
weight is given to the data at hand, the risk of overfitting occurs. The
resulting model will depend too much on the accidental peculiarities (or
noise) of the data. On the other hand, if too little weight is given, nothing will
be learned and the model completely relies on the prior knowledge. The right
balance between these opposite sides depends on the statistical significance
of the data. Obviously, the size of the data is an important factor. However,
the statistical significance also holds a relation with dimensionality.
Many of the mathematical techniques for modelling, tuning, training
and evaluation can be shared between the three subjects. Estimation
procedures used in classification can also be used in parameter estima-
tion or state estimation with just minor modifications. For instance,
probability density estimation can be used for classification purposes,
and also for estimation. Data-fitting techniques are applied in both