Page 169 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

158 CHAPTER 7 Pitfalls and Opportunities in the Development of AI Systems

With regard to the performance evaluation of AI systems, we have stressed the
necessity of rigorous evaluation methodology and the importance of the provision
of uncertainty along with performance metrics. We have also pointed to the defects
in prevalence dependent measures such as accuracy and the advantages of preva-
lence independent measures such as sensitivity, specificity, ROC, and AUC. Further
we have noted that emphasis on performance at any one particular operating point
on the ROC curve is misguided. Research has shown that operating points
can change radically between the development and implementation stages for AI
systems, but that AUC, for example, remains remarkably constant. For further
information on the AI development/evaluation process consult the appended
bibliography. Please note the extended tutorial on performance evaluation on our
website: davidgbrown.co [24].
It is not easy to participate in the responsible development and deployment of
AI systems; however, through rigorous attention to detail, honest evaluation, and
dedication to transparency, we can improve the present state of the process. If we
can improve the AI and diminish the AS, we can make the world a little less crappy
place. Go for it.

ACKNOWLEDGMENT
We thank Eugene O’Bryan for assisting us with his digital art skills.

REFERENCES

[1] N. Strausfeld, Arthropod Brains: Evolution, Functional Elegance, and Historical
Significance, Belknap Press, 2012.
[2] D. Fox, Consciousness in a cockroach, Discover Magazine (January 10, 2007).
[3] H.S. Mumby, S.N. Chapman, J.A. Crawley, K.U. Mar, W. Htut, A.T. Soe, et al., Distin-
guishing between determinate and indeterminate growth in a long-lived mammal, BMC
Evolutionary Biology 15 (2015) 214.
[4] National Health and Nutrition Examination Survey [Online]. 2013. Cited 2017 08 15.
Available from: https://wwwn.cdc.gov/Nchs/Nhanes/2013-2014/BMX_H.htm.
[5] E. Cohen, J. Conifield, cnn.com [Online]. 2016. Cited 2017 12 3. Available from: http://
www.cnn.com/2015/11/20/health/cancer-smelling-dogs/index.html.
[6] J. Fitzgerald, Artificial nose technology: status and prospects in diagnostics, The
American Journal of Human Genetics 35 (1) (January 2017) 33e42.
[7] J. Elmore, G. Longton, P. Carney, et al., Diagnostic concordance among pathologists
interpreting breast biopsy specimens, Journal of the American Medical Association
313 (11) (2015) 1122e1132.
[8] T. Cover, Geometrical and Statistical properties of systems of linear inequalities with
applications in pattern recognition, EC-14, IEEE Transactions on Electronic Computers
(1965) 326e334.