Page 37 -
P. 37
Why do we explicitly separate the dev set into Eyeball and Blackbox dev sets? Since you will
gain intuition about the examples in the Eyeball dev set, you will start to overfit the Eyeball
dev set faster. If you see the performance on the Eyeball dev set improving much more
rapidly than the performance on the Blackbox dev set, you have overfit the Eyeball dev set.
In this case, you might need to discard it and find a new Eyeball dev set by moving more
examples from the Blackbox dev set into the Eyeball dev set or by acquiring new labeled
data.
Explicitly splitting your dev set into Eyeball and Blackbox dev sets allows you to tell when
your manual error analysis process is causing you to overfit the Eyeball portion of your data.
Page 37 Machine Learning Yearning-Draft Andrew Ng
