Page 251 -

Exerciscs 239

Rosenblatt F (1962) Principles of Neurodynamics. Spartan Books. Washington DC.
Ruinelhart DE, Hinton GE, Williams RJ (1986) Learning Internal Representations by Error
Propagation. In: Rumelhart DE, McClelland JL (eds) Parallel Distributed Processing:
Explorations in the Microstructure of Cognition, vol.1, chapter 8, MIT Press.
Simon HU (1997) Bounds on the Number of Examples Needed for Learning Functions.
SIAM J. of Computing, 26:751-763.
Specht DF (1990) Probabilistic Neural Networks. Neural Networks, 13: 109-1 18.
Specht DF (1991) A Generalized Regression Neural Network. IEEE Tr Neural Networks,
21568-576.
Vapnik VN (1998) Statistical Learning Theory. Wiley, New York.
Van Rooij AJF, Jain LC, Johnson RP (1996) Neural Network Training Using Genetic
Algorithms. World Scientific Co. Pte. Ltd., Singapore.
Vonk E, Jain LC, Johnson RP (1997) Automatic Generation of Neural Network Architecture
Using Evolutionary Computation. World Scientific Co. Pte. Ltd., Singapore.
Weigend AS, Rumelhart DE, Huberman BA (1991) Generalization by Weight-Elimination
with Application to Forecasting. In: Lippman RP, Moody JE, Touretzky DS (eds)
Advances in Neural Information Processing Systems, 32375-882, Morgan Kaufmann,
California.
Widrow B, Glover Jr JR, McCool JM, Kaunitz J, Williams CS, Hearn RH, Zeidler JR, Dong
Jr E. Goodlin RC (1975) Adaptive Noise Cancelling: Principles and Applications Proc
IEEE, 63: 1692-1716.
Widrow B, Hoff Jr M (1960) Adaptive Switching Circuits. In: IRE WESCON Conv. Rec.,
4:96-104.

Exercises

5.1 Consider the adaptive noise cancelling application of a linear network, as in thc ECG
example described in section 5.1. Let R = E[xx'] represent the auto-correlation matrix
of the signal fed at the network inputs.
a) Compute the error expectation E[q], noticing that the error for the sample input
vector xi is q= $ - w'xj. Show that the Hessian matrix of the error energy is equal
to R.
b) Taking into account formula 5-42, prove that an upper bound of the learning factor
I], for the 50 Hz noise cancelling example of section 5.1 is q,,,, = 4/(n2~, where a
is the amplitude of the sinusoidal inputs. Check this upper bound using the
ECG5OHz.xls file.
c) The time constant of an exponential approximating the mean-square error learning
curve is given by r=w1(41] tr(R)), where tr(R) is the sum of the eigenvalues of R.
Show that for the pure sinusoidal noise cancelling example in section 5.1, the time
constant is r=1/(2na2q).

5.2 Determine the optimal parabolic discriminants for the two-class one-dimensional
problems shown in Figure 5.8 and Figure 5.9 by solving the respective normal
equations.

5.3 Implement the first two steps of the gradient descent approach for the parabolic
discriminant of the two-class one-dimensional problems shown in Figure 5.9, starting