Page 173 - Computational Retinal Image Analysis

168 CHAPTER 9 Validation

5 Conclusion

Unless somebody like you / Cares a whole awful lot / Nothing is going to get bet-
ter; / It’s not.
Dr. Seuss, The Lomax
The definition and practice of validation of medical image analysis algorithms and
software must adapt to the transformations brought about by deep learning and sup-
porting hardware, enabling analysis of larger and larger volumes of data. This chal-
lenges the conventional paradigm of specialists annotating images in detail, triggering
the need for effective techniques limiting the amount of annotations required to train a
model. At the same time, research has exposed the pitfalls of the conventional wisdom
of comparing annotations with the output of programs. The quantitative effect of uncer-
tainty on measurements from image analysis on the results of statistical studies aimed
at biomarker discovery, among others, remains unclear. We feel that these issues, taken
together, are at the very core of the difference between reliable and unreliable science.
How can validation practice be improved to facilitate the translation of novel,
reliable technology into healthcare? Part of the answer are visible international con-
sortia promoting good practice in technical challenges with large, public data sets;
establishing, as much as possible, agreement on validation definition, criteria, prac-
tice and protocols across different challenges and groups in different domains (e.g.,
automatic analysis of images from radiology or ophthalmology, from surgery-related
instruments, etc.); the consequent generation of recommendations approved by inter-
national, high-quality consortia of visible groups from both the image/data process-
ing and medical disciplines; ideally, and ultimately, the generation of international
standards approved by relevant regulatory bodies.

Acknowledgments

A. Doney, S. Hogg, M.R.K. Mookiah and E. Trucco acknowledge funding from the National
Institute for Health Research (INSPIRED: India-Scotland grant on precision medicine for dia-
betes). A. McNeil acknowledges funding from Medical Research Scotland. L. Ballerini, A.
Doney, T. MacGillivray, S. McGrory and E. Trucco acknowledge funding from the Engineering
and Physical Science Research Council (EP/M005976/1). L. Ballerini acknowledges funding
from LBC1936 Age UK, the Medical Research Council (MR/M01311/1), the Fondation Leducq
(Network for the Study of Perivascular Spaces in Small Vessel Disease, 16 CVD 05) and the
UK Dementia Research Centre at the University of Edinburgh. T. MacGillivray acknowledges
funding from the Edinburgh Clinical Research Facility at the University of Edinburgh.

References
[1] P. Jannin, C. Grova, C.R. Maurer, Model for defining and reporting reference-based
validation protocols in medical image processing, Int. J. Comput. Assist. Radiol. Surg. 1
(2006) 63–73.