Page 50 -

24 Bias vs. Variance tradeoff

You might have heard of the “Bias vs. Variance tradeoff.” Of the changes you could make to
most learning algorithms, there are some that reduce bias errors but at the cost of increasing
variance, and vice versa. This creates a “trade off” between bias and variance.

For example, increasing the size of your model—adding neurons/layers in a neural network,

or adding input features—generally reduces bias but could increase variance. Alternatively,
adding regularization generally increases bias but reduces variance.

In the modern era, we often have access to plentiful data and can use very large neural
networks (deep learning). Therefore, there is less of a tradeoff, and there are now more
options for reducing bias without hurting variance, and vice versa.

For example, you can usually increase a neural network size and tune the regularization
method to reduce bias without noticeably increasing variance. By adding training data, you
can also usually reduce variance without affecting bias.

If you select a model architecture that is well suited for your task, you might also reduce bias
and variance simultaneously. Selecting such an architecture can be difficult.

In the next few chapters, we discuss additional specific techniques for addressing bias and
variance.

Page 50 Machine Learning Yearning-Draft Andrew Ng