Page 168 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

5. Conclusion 157

FIGURE 7.13
Uncertainty estimates of our CI’s performance as a function of the training sample size
[23]. Mean jackknife estimates are open circles and solid circles are the mean bootstrap
estimates of uncertainty due to the variation in the training dataset. The solid line is the
true standard deviation of performance. As the number of patients in the training sample
increases, the variability of the CI’s performance on the population decreases, and the
jackknife and bootstrap are good methods of estimating this uncertainty.

performance of CI due to the training sample is shown in Fig. 7.13. The plot also
shows the mean bootstrap and jackknife estimates of this variability.

5. CONCLUSION

In the development of AI systems there are many traps for the unwary. We have out-
lined our concerns about the data used to train and test these systems, the architec-
ture of the systems themselves, and the techniques used in their evaluation. The data
used should be chosen to be the most internally consistent, least noisy, highest qual-
ity, and most relevant to your task. Our algorithm should not be driven too hard to
achieve the best possible result on the training set: the benefits from this are illusory,
as generalizability is being sacrificed. In addition, good data hygiene must be main-
tained in not reusing data for testing that has already been employed during the
training phase. These facets should all be examined carefully and either remediated
or fully disclosed. We should do what we can and disclose the rest.