Page 323 - Classification Parameter Estimation & State Estimation An Engg Approach Using MATLAB

312 WORKED OUT EXAMPLES

The five-fold cross-validation errors of ldc and qdc are 13.0% and
17.4%, respectively. Note that by using the function crossval,we
avoided having to use the training set for testing. If we had done that,
the errors would be 11.9% and 16.2%, but this estimate would be
biased and too optimistic. The results also depend on how the data is
randomly split into batches. When this experiment is repeated, you
will probably find slightly different numbers. Therefore, it is advisable
to repeat the entire experiment a number of times (say, 5 or 10) to get
an idea of the variation in the cross-validation results. However, for
many experiments (such as the feature selection and neural network
training below), this may lead to unacceptable training times; there-
fore, the code given does not contain any repetitions. For all the
results given, the standard deviation is about 0.005, indicating that
the difference between ldc and qdc is indeed significant. Notice that
we use the word ‘significant’ here in a slightly loose sense. From a
statistical perspective it would mean that for the comparison of the
two methods, we should state a null-hypothesis that both methods
perform the same, and define a statistical test (for instance a t-test) to
decide if this hypothesis holds or not. In this discussion we use the
simple approach in which we just look at the standard deviation of
the classifier performances, and call the performance difference sig-
nificant when the averages differ by more than two times their stand-
ard deviations.
Even with these simple models on the raw, not preprocessed data, a
relative good test error of 13.0% can be achieved (with respect to the
simplest approach by looking at the class probabilities). Note that qdc,
which is a more powerful classifier, gives the worst performance. This is
due to the relatively low sample size: two covariance matrices, with
1 13(13 þ 1) ¼ 91 free parameters each, have to be estimated on 80%
2
of the data available for the classes (i.e. 172 and 233 samples, respect-
ively). Clearly, this leads to poor estimates.

9.1.3 Feature extraction

It might be expected that more flexible methods, like the -nearest neigh-
bour classifier (knnc, with optimized using the leave-one-out method)
or the Parzen density estimator (parzenc) give better results. Surpris-
ingly, a quick check shows that then the errors become 19.6% and 21.9%
(with a standard deviation of about 1.1% and 0.6%, respectively), clearly