Page 81 - Handbook of Deep Learning in Biomedical Engineering Techniques and Applications

Chapter 3 Application, algorithm, tools directly related to deep learning 69

3. Algorithms

3.1 Deep belief networks
Deep belief networks (DBNs) are graphical models used to
extract a deep hierarchical representation of the training data.
They model the joint distribution parameters between observed
k
vector x and the [ hidden layers h as Eq. 3.1 [12]:
!
[ 2
1

P h jh
P x; h ; .; h [ ¼ Y k kþ1 P h [ 1 ; h [ (3.1)
k ¼ 0
0
h
where x ¼ h , p h k 1 k is a conditional distribution for the all

visible units conditioned on the hidden units of the restricted
Boltzmann machine (RBM) at levelk, and P h [ 1 ; h [ is the

visible-hidden joint distribution in the top-level RBM. This is
shown in Fig. 3.4 [12].
The main principle of greedy layer-wise unsupervised training
can be applied to all DBNs [13]. The process is considered as
follows:
1. Train the initial layer as an RBM that models the raw input
x ¼ h ð0Þ as its visible layer.
2. Use that first layer to obtain a representation of the input that
will be fed back as data for the second layer. There exist two
common solutions. This representation can be shown as being

the mean activations p h ð1Þ ¼ 1 h or samples of

ð0Þ

p h ð1Þ ð0Þ .
h

h 3
RBM

h 2

h 1

x
Figure 3.4 Visible joint distribution. From http://deeplearning.net/tutorial/DBN.
html.