Page 242 -

240 R. Seri and D. Secchi

11.4.2.1 Identifying the Appropriate Effect Size

We are left with the problem of guessing a value for the effect size. In the following
we propose some reasonings concerning the choice. For this purpose, we need some
m (Cohen

basic notation. For ANOVA, the effect size f is to be calculated as f D
1992), where m is the standard deviation of the group means and is the within-
population (or pooled) standard deviation. The same quantity can be expressed
using sums of squares. Let the subscript j, ranging from 1 to G, indicate the j-
th group/CoP. Let n be the number of observations in each group/CoP and let i,
ranging from 1 to n, denote the i-th observation. Therefore, x ij is the i-th observation
1 P n
x
n
in the j-th group. Moreover, we consider the sample means Nx j D iD1 ij and
1 P n P G
x
nG
N x D iD1 jD1 ij , where the latter is the grand mean of all the observations.
Now, the variation of the observations between configurations (or groups) is simply
P G 2
the square root of the Sum of Squares Between (SSB), or n N x j Nx , divided
jD1
P G P n 2
by the Sum of Squares Within (SSW) observations, or jD1 iD1 x ij Nx j .
A first possibility is to use “canned” effect sizes (Cohen 1988). In this case,
verbal descriptions of the strength of the effect (e.g., small, medium and large; see
below) are mapped onto numerical values, usually based on effect sizes retrieved
from a review of the literature. As per the literature on the GCM, we can expect
that many decisions are made by oversight and that the number of difficult problems
solved increases under anarchy (Fioretti and Lomi 2008, 2010; Herath et al. 2015).
Hence, the “distance” between conditions could be classified as medium or large,
and we can set to an effect size of 0:25 or 0:4 (consistently with Cohen 1992). The
computation of the sample size providing the desired level of power can use the
formulas in Cohen (1988) as implemented in the R package pwr on power analysis
(see Champely et al. 2016). For f D 0:25, this yields the result n D 112 (exactly

111:68). A simpler approximation, proposed in Secchi and Seri (2017), yields n D
12
109 (exactly 109:47). The latter approach makes the relation between components
of power more explicit. For f D 0:40, R package pwr yields n D 45 (more precisely
44:58) and our formula yields n D 43 (more precisely 43:04).
A second possibility is to guess a value for the effect size by having the
simulation run for a pilot study and by calculating the estimated effect size from
the results. We will deal below with some problems involved in this approach.
For expository purposes, we have decided to run the model for n D 10 runs per
condition. Taking the definition of f above, the numerator SSB is 0:000813 and the
denominator is 0:004341 so that the estimated effect size for these three conditions
is 0:43. If we calculate the number of runs reaching a power of 0:95 with such
a large effect size as f D 0:43, we obtain n D 38 (more precisely 38:31)using
Cohen’s formulas and n D 37 (more precisely 36:81) using our approximation.
A third possibility is to use the results of former studies on the same topic.
Here, we can use the results of Fioretti and Lomi (2010) in the case without buck

12 This formula can be used in R with an ad hoc function taken from one of our previous publications
(Secchi and Seri 2017). See the Appendix for the code for both formulas.