Page 162 -

8 The Importance of Ontological Structure: Why Validation by ‘Fit-to-Data’... 159

to context, suggesting there is no universally applicable trade-off heuristic. In the
meantime, in the social simulation community, common practice is either to use
stakeholder evaluation in participatory contexts or simply to rely on peer review.
Given that ontological expressivity is a major advantage for agent-based modelling,
at least as suggested by our categorizations in Appendix 3, the community should
set itself the aim of finding ways to quantify that benefit.
We emphasize that our arguments do not mean that fit-to-data can be ignored
as a criterion for assessing the confidence we should have but rather that we also
need to pay attention to the model’s ontology. As this section has shown, there are
various ways to do this, though the area is far from being sufficiently settled that
we can provide ‘generally used’ quantitative measures of the fit of an ontology to
a system. This may reflect the fact that agent-based simulation, a relatively recent
development in the world of modelling, has a much greater potential expressivity
in its ability to specify ontologies, and the question of model structure has thus far
been limited to discussions about numbers of parameters. Further, there is evidence
that we should not expect to find a single, general measure that appropriately trades
off fit-to-data and ontological fit and provides us with a number that tells us how
‘good’ a model is.

8.1.6 Conclusion

Summarizing the key arguments in this chapter:
• Methods for validating models have thus far concentrated on fit-to-data.
• There are various ways in which that fit can be assessed.
• However, fit-to-data, though it should not be ignored, cannot be trusted as the sole
basis for model validation. Besides questions about comparability of context, the
modeller’s biases in encoding, or representing, the system need to be questioned.
• In complex open systems, fit-to-data does not resolve whether a model’s
predictions should be trusted.
• Agent-based models have greater potential ontological expressivity than other
modelling approaches, and researchers wishing to validate their models need to
pay attention to their ontology as well as their fit-to-data.
The effort involved in building an agent-based model in an empirical context,
as opposed to a more traditional aggregate-level mathematical model, is predicated
on the empirical world being ‘complex’. In such systems, validation by fit-to-data
is not, on its own, a sound basis for estimating the ability of a model to make
reliable predictions, not least because of issues with path dependency. However, the
availability of sufficient data to justify building a model with as many parameters
as an agent-based model typically has is a further significant potential issue, at
least until methods are developed to assess how flexible agent-based models are
in their ability to realize input-output mappings. These points, however, apply just