Page 50 -

44 B. Edmonds

data the first time the model was tested, it was then adjusted until it did work, or,
alternatively, only those models that fitted the out-of-sample data were published (a
publishing bias). Thus, in these cases, the models were not tested against predicting
the out-of-sample data even though they were presented as such. Fitting known data
is simply not a sufficient test for predictive ability.
There are many reasons why prediction of complex social systems fails, but two
of the most prominent are (1) it is unknown what processes are needed to be included
in the model and (2) a lack of enough quality data of the right kinds. We will discuss
each of these in turn.
1. In the physical sciences, there are often well-validated micro-level models (e.g.
fluid dynamics in the case of weather forecasting) that tell us what processes are
potentially relevant at a coarser level and which are not. In the social sciences,
this is not the case—we do not know what the essential processes are. Here, it is
often the case that there are other processes that the authors have not considered
that, if included, would completely change the results. This is due to two different
causes: (a) we simply do not know much about how and why people behave
in different circumstances, and (b) different limitations of intended context will
mean that different processes are relevant.
2. Unlike in the physical sciences, there has been a paucity of the kind of data we
would need to check the predictive power of models. This paucity can be due
to (a) there is not enough data (or data from enough independent instances) to
enable the iterative checking and adapting of the models on new sets of unknown
data each time we need to, or (b) the data is not of the right kind to do this. What
can often happen is that one has partial sets of data that require some strong
assumptions in order to compare against the predictions in question (e.g. the data
might only be a proxy of what is being predicted, or you need assumptions in
order to link sets of data). In the former case, (a), one simply has not enough to
check the predictive power in multiple cases, so one has to suspend judgement as
to whether the model predicts in general, until the data is available. In the latter
case, (b), the success at prediction is relative to the assumptions made to check
the prediction.
A more subtle risk is that the conditions under which one can rely upon a model
to predict well might not be clear. If this is the case, then it is hard to rely upon the
model for prediction in a new situation, since one does not know its conditions of
application.

4.2.3 Mitigating Measures

To ensure that a model does indeed predict well, one can seek to ensure the
following:
• That the model has been tested on several cases where it has successfully
predicted data unknown to the modeller (at the time of prediction)