Page 153 -

150 G. Polhill

coloured curves show the values of n (on the x-axis) at which P in (8.1) has an
upper bound of 0.001. For example, if d VC D 2 (cyan curve), and " D 0.05, then
5
n needs to be roughly 10 for P to have an upper bound of 0.001. For quantitative
social data, that would be a very simple model for a very expensive questionnaire.
These high estimates are partly a consequence of the fact that the VC formula
and growth function m() are both upper bounds. However, the high estimates are
also a consequence of the function under scrutiny essentially being an arbitrary
choice, without any other information about the system the data have come from
or the way the model describes that system. The VC formula is therefore very
much a ‘worst case’, but one that applies to neural networks insofar as relatively
little information about the system is encoded in the network’s topology. That
information is essentially the modeller’s assumptions about the appropriate level
of ‘wiggliness’ needed to fit the data – which may be as much about the pragmatics
of training the network and the amount of data available as it is a reflection of the
system the data have come from.
Using knowledge to constrain the choice of model is one way to reduce the
VC estimate. Traditionally, this might be achieved effectively by reducing the
VC dimension of the set of models being considered, using the kind of practice
criticized above for being ‘unscientific’. Introducing bias by removing variables
from consideration, reducing the number of parameters on terms using those
variables (e.g. by only considering linear models) or making other oversimplifying
assumptions is, however, not the only way that we can constrain our choice of
model. Though the impact on the VC dimension is less clear, in agent-based
models, we can also constrain our choice of model by making it more ‘descriptive’
(Edmonds and Moss 2005). This essentially amounts to appropriately tuning the
model’s ‘ontology’ or ‘microworld’, but before considering the ontology in more
detail, since agent-based models are typically applied to complex systems, we will
consider some arguments about validation by fit-to-data in such systems.

8.1.4 Complex Systems and Validation by Fit-to-Data

Since agent-based models are applied to complex systems, this section introduces an
important article (Oreskes et al. 1994) posing arguments about the degree to which
we should trust fit-to-data as a measure of our confidence in a model’s predictions in
complex open systems. We move on to criticize Ockham’s razor – a heuristic often
used by modellers to give preference to simpler models with the same fit-to-data and
one that has already been argued against on different grounds by Edmonds (2002).
Naomi Oreskes et al. (1994) have argued eloquently that environmental systems
(and hence socio-environmental systems) are ‘open’, and hence traditional valida-
tion expressed as fit-to-data commits a logical fallacy when used as a basis to judge
the degree of belief we should have that a model is a ‘good’ one. Essentially, the
fallacious argument affirms the consequent by starting with the observations that