Page 91 - Handbook of Deep Learning in Biomedical Engineering Techniques and Applications

Chapter 3 Application, algorithm, tools directly related to deep learning 79

Disadvantages of RNN:
1. Gradient vanishing problems and exploding problems.
2. Training an RNN for a computational problem is a very tedious
task.
3. It cannot execute very long sequences if tan h is used as an
activation function.
3.4 Long short-term memory networks

LSTM is a one kind of RNN. In RNN, output from the last step is
fed as input to the current step. LSTM was designed by Hochreiter
and Schmidhuber. It tackled all the problems of long-term depen-
dencies of RNN in which the RNN cannot estimate the word stored
in the long-term memory, but they can give more accurate predic-
tions from the recent information. Since the gap length increases,
RNN does not give efficient results [19].
3.4.1 Structure of long short-term memory
LSTM has a chainlike structure that contains four neural
networks and different kinds of memory blocks called cells. The
structure of LSTM is illustrated in Fig. 3.15 [31].
Information is retained only by the cells, and the memory ma-
nipulations are performed by the gates [19]. There are three gates:
Forget gate: The information that no longer used in the cell
state is pullout with the forget gate. Two inputs x(t) and h(t 1)
are fed to the gate and multiplied with their corresponding weight
matrices and then followed by the addition of tiny bias value.

Figure 3.15 Structure of LSTM. From https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:The_
LSTM_cell.png.