Page 156 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

2. AI Development 145

development of a navigation aide for Dr. Strange in his bizarre world. Our data
must accord with the task we are using it for.
We don’t know “ground truth” for our casesdwe never do. Unless our data is
simulateddand thereby not directly applicable to real-world problemsdwe always
have a problem knowing the validity of assignment of cases to our classes. Well,
perhaps that’s not much of a problem for distinguishing elephants from people; how-
ever, I’ll bet that for pet images from the internet, some of our dogs would meow. In
the medical context we often use human experts for ground truth, and most people
would readily agree that radiologists can make mistakes in the assignment of normal
versus lesion present cases from radiographs. What is more startling is the propensity
for disagreement among pathologists, see Fig 7.6. If we use pathology to determine
ground truth, then for the sake of our equanimity we might use one pathologist rather
than a panel; because, if we do the latter, we’ll discover how frequently they disagree.
Of course, this is not a recommendation, because we need to use the best possible
estimate for our ground truth.
This leads to a second “ground truth” concern. Obviously there are easier and
tougher cases for any discrimination task, and clearly the uncertainty in ground truth
is directly correlated with the degree of case difficulty.
Our data are noisy. All data are noisy, especially if they come from sensors of
some kind. This at least should be quantifiable. If the data are mediated by human
intervention, then additional variation is to be expected. We must try to identify
and where possible minimize the noise in our data to avoid propagating error in
our features and final outcome.
Our data are incomplete. Images can be misplaced; questionnaire items can be
left blank; data items can be “out of range” and nonsensical. For each of our data
points or cases we need to be sure that all appropriate data are present or we must
have a justifiable procedure in place to handle any deficits.

FIGURE 7.6
A panel of three typical pathologists. We are particularly concerned by the results reported
by the one on the left. For pathology discordance see, for example, Elmore et al. [7],
finding that “Among the 3 consensus panel members, unanimous agreement of their
independent diagnoses was 75%.”
Image Garosha j Dreamstime.