Page 326 - Classification Parameter Estimation & State Estimation An Engg Approach Using MATLAB

BOSTON HOUSING CLASSIFICATION PROBLEM 315

% Calculate cross-validation error for classifiers
% trained on the optimal 5-feature set
err_ldc_fsf ¼ crossval(z,w_fsf*ldc,5)
err_qdc_fsf ¼ crossval(z,w_fsf*qdc,5)
err_knnc_fsf ¼ crossval(z,w_fsf*knnc,5)
err_parzenc_fsf ¼ crossval(z,w_fsf*parzenc,5)

The feature selection routine often selects features STATUS, AGE and
WORK, plus some others. The results are not very good: the perform-
ance decreases for all classifiers. Perhaps we can do better if we take the
performance of the actual classifier to be used as a criterion, rather than
the general inter–intra class distance. To this end, we can just pass the
classifier to the feature selection algorithm. Furthermore, we can also let
the algorithm find the optimal number of features by itself. This means
that branch-and-bound feature selection can now no longer be used, as
the criterion is not monotonically increasing. Therefore, we will use
forward feature selection, featself.

Listing 9.5

% Load the housing dataset
load housing.mat;
% Optimize feature set for ldc
w_fsf ¼ featself([],ldc,0)*scalem([],‘variance’);
err_ldc_fsf ¼ crossval(z,w_fsf*ldc,5)
% Optimize feature set for qdc
w_fsf ¼ featself([],qdc,0)*scalem([],‘variance’);
err_qdc_fsf ¼ crossval(z,w_fsf*qdc,5)
% Optimize feature set for knnc
w_fsf ¼ featself([],knnc,0)*scalem([],‘variance’);
err_knnc_fsf ¼ crossval(z,w_fsf*knnc,5)
% Optimize feature set for parzenc
w_fsf ¼ featself([],parzenc,0)*scalem([],‘variance’);
err_parzenc_fsf ¼ crossval(z,w_fsf*parzenc,5)

This type of feature selection turns out to be useful only for ldc and
qdc, whose performances improve to 12.8% ( 0:6%) and 14.9%
( 0:5%), respectively. knnc and parzenc, on the other hand, give
15.9% ( 1:0%) and 13.9% ( 1:7%), respectively. These results do
not differ significantly from the previous ones. The featself routine
often selects the same rather large set of features (from most to least
significant): STATUS, AGE, WORK, INDUSTRY, AA, CRIME,
LARGE, HIGHWAY, TAX. But these features are highly correlated,
and the set used can be reduced to the first three with just a small