Page 139 -

126 4 Statistical Classification

of the two closest classes as a merit criterion. Furthermore, features were only
entered or removed from the selected set if they contributed significantly to the
Anova F. The solution corresponding to Figure 4.37 used a 5% level for the
statistical significance of a candidate feature to enter the selected set and 10% to
remove it. Notice that PRT, which had entered at step 1, was later removed, at step
5. The nested solution {PRM, N, ARTG, RAAR) would not have been found by a
direct forward search.

4.5 Classifier Evaluation

The determination of reliable estimates of a classifier error rate is obviously an
essential task in order to assess its usefulness and to compare it with alternative
solutions.
As explained in section 4.2.3 design set estimates are on average optimistic and
the same can be said about using an error formula such as (4-25), when true means
and covariance are replaced by their sample estimates. It is, therefore, mandatory
that the classifier be empirically tested, using a test set of independent cases. As
mentioned already in section 4.2.3, these test set estimates are on average
pessimistic.
We describe in the following the influence of the finite sample sizes of the
design and test sets on the classifier performance. For this purpose, we consider a
two-class classifier with Bayes error:

The influence of the finite sample sizes can be summarized as follows (for
details, consult Fukunaga, 1990).

Influence of finite test set

Let Pe,(n) be the test set estimate, influenced only by the finiteness of the test set,
and consider the ensemble average of all such estimates, E[Pe,(n)], of a given
classifier with Bayes error Pe. The expectation E[Pe,(n)] can be computed with
arbitrarily large accuracy for a growing number of these estimates, with
independent sets of size n. The following results for the expectation and variance
are verified:

Therefore, test set estimates are unbiased, but have a variance inversely
proportional to the number of test samples (n, for w, and n2 for q).