Page 148 -

8 The Importance of Ontological Structure: Why Validation by ‘Fit-to-Data’... 145

Bearing this in mind, at the end of the calibration process, you have a param-
eterized neural network with all the weights specified that you now want to be
able to use to make predictions with; except, of course, if you want to have some
degree of confidence in those predictions. Validation is the process of developing
that confidence, and it is achieved by using the data you kept aside and didn’t use
during calibration to estimate how good your future predictions will be. So, having
reached a point where you are happy with the error on the calibration data, you use
the validation data to tell you how confident you should be in the model you have
fitted: the error rate on the validation set is an estimate of the expected error rate for
prediction.
Generalization is the ability of the model to provide output for untrained input.
There are two aspects to this. The first is whether the required input can be
represented using the formalism provided by the model. In the case of neural
networks, the question seems simply to be whether the input can be adequately
expressed using the same set of dimensions and any encoding thereof as the data
used for calibration and validation. It may seem unfair to expect a model to be able
to provide output for cases that cannot be expressed using the ‘language’ the model
was built with. However, sometimes, arguably, that is what happens. Measures of
inflation, for example, are based on a ‘basket of goods’ that changes from time to
time as people’s buying habits change. This change arguably changes the meaning
of inflation. Though something of a straw man, if you have calibrated a model using
a measure of inflation that uses one basket of goods and then naively expect it to
give meaningful output for a measure of inflation that uses another, then perhaps
you are expecting the model to provide output for cases that cannot be expressed
1
using the language the model was built with. Similar problems exist with other
social statistics that might be used as variables input to or output from a model,
particularly where there are changes in the way the variables are measured from one
region to another.
A second problem comes from what you left out of the model when you first built
it. Although this too may seem like an unfair criticism, perhaps when you built the
original model, a particular variable was not included as an input variable because
it was not seen as having any significant relationship with the output. Since the
model was calibrated and validated, however, a large change in the ignored variable
might have occurred that has affected the relationships between the variables you
did include. So, although when you come to compute a prediction for a new input
you have all the data you need, and can perform the computation, really, the values
for the variables you have as inputs to your model do not adequately reflect the
scenario any more. This is known as ‘omitted variable bias’ in the econometrics
literature (see, e.g. Clarke 2005).

1
Less naively, you would use a calculated inflation figure for the old basket of goods as input to
the model; however, if people are not buying things in the old basket, the model may still not be
providing meaningful output.