Page 318 - Introduction to Statistical Pattern Recognition

Chapter 7

NONPARAMETRIC CLASSIFICATION

AND ERROR ESTIMATION

After studying the nonparametric density estimates in Chapter 6, we are
now ready to discuss the problem of how to design nonparumetric clussifiers
and estimate their classification errors.

A nonparametric classifier does not rely on any assumption concerning
the structure of the underlying density function. Therefore, the classifier
becomes the Bayes classifier if the density estimates converge to the true den-
sities when an infinite number of samples are used. The resulting error is the
Bayes error, the smallest achievable error given the underlying distributions.
As was pointed out in Chapter 1, the Bayes error is a very important parameter
in pattern recognition, assessing the classifiability of the data and measuring
the discrimination capabilities of the features even before considering what
type of classifier should be designed. The selection of features always results
in a loss of classifiability. The amount of this loss may be measured by com-
paring the Bayes error in the feature space with the Bayes error in the original
data space. The same is true for a classifier. The performance of the classifier
may be compared with the Bayes error in the original data space. However, in
practice, we never have an infinite number of samples, and, due to the finite
sample size, the density estimates and, subsequently, the estimate of the Bayes
error have large biases and variances, particularly in a high-dimensional space.

300