Page 49 -

4 Different Modelling Purposes 43

which involves the long-term development of simulations that carefully see what
can be inferred from the available data. As well as making predictions, his unit tries
to establish the level of uncertainty in those predictions—being honest about the
probability of those predictions coming about given the likely levels of error and
bias in the data. As described in his book (Silver 2012), this involves a number of
properties and activities, including:

• Repeated testing of the models against unknown data
• Keeping the models fairly simple and transparent so one can understand clearly
what they are doing (and what they do not cover)
• Encoding into the model aspects of the target phenomena that one is relatively
certain about (such as the structure of the US presidential electoral college)
• Being heavily data biased, requiring a lot of data to help eliminate sources of
error and bias
• Producing probabilistic predictions, giving a good idea about the level of
uncertainty in any prediction
• Being clear about what kinds of factors are not covered in the model, so the
predictions are relative to a clear set of declared assumptions and one knows the
kind of circumstances in which one might be able to rely upon the predictions
Post hoc analysis of predictions—explaining why it worked or not—is kept
distinct from the predictive models themselves; this analysis may inform changes to
the predictive model but is not then incorporated into the model. The analysis is thus
kept independent of the predictive model, so it can be an effective check. Making
a good predictive model requires a lot of time getting it wrong with real, unknown
data and trying again before one approaches qualified successful predictions.

4.2.2 Risks

Prediction (as we define it) is very hard for any complex social system. For this
7
reason, it is rarely attempted. Many re-evaluations of econometric models against
data that has emerged since publication have revealed a high rate of failure (e.g.
Meese and Rogoff 1983)—37 out of 40 models failed completely. Clearly, although
presented as being predictive models, they did not actually predict unknown data.
Many of these used the strategy of first dividing the data into in-sample and out-of-
sample data, and then parameterising the model on the former and exhibiting the fit
against the latter. Presumably, the apparent fit of the 37 models was not simply a
matter of bad luck, but that all of these models had been (explicitly or implicitly)
fitted to the out-of-sample data, because the out-of-sample data was known to the
modeller before publication. That is, if the model failed to fit the out-of-sample

7 To be precise, some people have claimed to predict various social phenomena, but there are very
few cases where the predictions are made public before the data is known and where the number of
failed predictions can be checked. Correctly predicting events after they are known is much easier!