Page 74 - Handbook of Deep Learning in Biomedical Engineering Techniques and Applications

62 Chapter 3 Application, algorithm, tools directly related to deep learning

Figure 3.1 Concept of deep learning. Adapted from https://www.tutorialspoint.com/tensorflow/tensorflow_quick_
guide.htm.

Figure 3.2 Graph using sigmoid function. From https://en.wikipedia.org/wiki/File:Logistic-curve.svg.

The value of slope or the gradient of this function, at the
extreme ends, is always close to zero. Therefore, the parameters
are updated in a very slow manner, resulting in very slow
learning. Hence, exchanging from a sigmoid activation function
to ReLU (rectified linear unit) is one of the biggest progresses
we have seen in neural networks. ReLU updates the parameters
much faster when the value of slope is 1 when x > 0. This is the
dominant reason for faster computation of the models [1].