Page 138 -

7 Checking Simulations: Detecting and Avoiding Errors and Artefacts 135

Modeller’s activities:
• Develop and analyse new formal models by implementing alternative accessory
assumptions while keeping the core assumptions identified by the thematician.
This exercise will help to detect artefacts. Only those conclusions which are
not falsified by any of these models will be valid for the thematician’s model.
As an example, see Galán and Izquierdo (2005), who studied different instan-
tiations of one single conceptual model by implementing different evolutionary
selection mechanisms. Takadama et al. (2003) conducted a very similar exercise
implementing three different learning algorithms for their agents. In a collection
of papers, Klemm et al. (2003a, 2003b, 2003c, 2005) investigate the impact
of various accessory assumptions in Axelrod’s model for the dissemination of
culture (Axelrod 1997b). Another example of studying different formal models
that address one single problem is provided by Kluver and Stoica (2003).
• Conduct a more exhaustive exploration of the parameter space within the
boundaries of the thematician’s specifications. If we obtain essentially the same
results using the wider parameter range, then we will have broadened the scope of
the model, thus making it more representative of the thematician’s model. If, on
the other hand, results change significantly, then we will have identified artefacts.
This type of exercise has been conducted by, e.g. Castellano et al. (2000) and
Galán and Izquierdo (2005).
• Create abstractions of the formal model which are mathematically tractable. An
example of one possible abstraction would be to study the expected motion of
a dynamic system (see the studies conducted by Galán and Izquierdo (2005),
Edwards et al. (2003), and Castellano et al. (2000) for illustrations of mean-field
approximations). Since these mathematical abstractions do not correspond in a
one-to-one way with the specifications of the formal model, any results obtained
with them will not be conclusive, but they may suggest parts of the model where
there may be errors or artefacts.
• Apply the simulation model to relatively well-understood and predictable sit-
uations to check that the obtained results are in agreement with the expected
behaviour (Gilbert and Terna 2000).
Computer scientist’s activities:
• Develop mathematically tractable models of certain aspects, or particular cases,
of the modeller’s formal model. The analytical results derived with these models
should match those obtained by simulation; a disparity would be an indication of
the presence of errors.
• Develop new executable models from the modeller’s formal model using alter-
native modelling paradigms (e.g. procedural vs. declarative). This activity will
help to identify artefacts. As an example, see Edmonds and Hales’ (2003)
reimplementation of Riolo et al. (2001) model of cooperation among agents using
tags. Edmonds reimplemented the model using SDML (declarative), whereas
Hales reprogrammed the model in Java (procedural).