Page 69 -

35 Surpassing human-level performance

You are working on speech recognition and have a dataset of audio clips. Suppose your

dataset has many noisy audio clips so that even humans have 10% error. Suppose your
system already achieves 8% error. Can you use any of the three techniques described in
Chapter 33 to continue making rapid progress?

If you can identify a subset of data in which humans significantly surpass your system, then
you can still use those techniques to drive rapid progress. For example, suppose your system
is much better than people at recognizing speech in noisy audio, but humans are still better

at transcribing very rapidly spoken speech.

For the subset of data with rapidly spoken speech:

1. You can still obtain transcripts from humans that are higher quality than your algorithm’s
output.

2. You can draw on human intuition to understand why they correctly heard a rapidly
spoken utterance when your system didn’t.

3. You can use human-level performance on rapidly spoken speech as a desired performance
target.

More generally, so long as there are dev set examples where humans are right and your

algorithm is wrong, then many of the techniques described earlier will apply. This is true
even if, averaged over the entire dev/test set, your performance is already surpassing
human-level performance.

There are many important machine learning applications where machines surpass human
level performance. For example, machines are better at predicting movie ratings, how long it
takes for a delivery car to drive somewhere, or whether to approve loan applications. Only a

subset of techniques apply once humans have a hard time identifying examples that the
algorithm is clearly getting wrong. Consequently, progress is usually slower on problems
where machines already surpass human-level performance, while progress is faster when
machines are still trying to catch up to humans.

Page 69 Machine Learning Yearning-Draft Andrew Ng