Page 146 -

8 The Importance of Ontological Structure: Why Validation by ‘Fit-to-Data’... 143

8.1.1 Introduction to Neural Networks

In this section, we will briefly explain what neural networks are, the mathematical
formulas that underpin them (in Appendix 1) and the way they are structured.
The main points we wish to introduce are that, though neural networks have
tremendous potential to approximate data, there is nothing about their structure
or the mathematics underpinning their functioning that necessarily reflects any
structure or mathematics in whatever system the data were taken from.
Neural networks were originally conceived as simulations of the brains but
are essentially networks of nonlinear functions with parameters that are adjusted
according to a learning rule. There are several different kinds of neural network
mathematically speaking, and for each kind, there can be several different learning
rules and minor adaptations and variations thereof. Biologically, a neuron is a cell
with axons connecting it to other neurons. In an agent-based simulation of a brain,
we would simulate a neuron as an agent and an axon as a link. The behaviour of
the neuron is simply to emit an electrical pulse periodically. The more frequent
the pulse, the more ‘excited’ the neuron. Connections between neurons can be
excitatory or inhibitory. An excitatory connection means that there is a positive
relationship between the excitation of the two connected neurons: all other things
being equal, one neuron’s excitement increases that of the other. An inhibitory
connection means that the relationship is negative – one neuron’s excitement
decreases that of the other. The connection has a strength – the stronger the
connection between one neuron and another, the more significant the relationship
is in comparison with other neurons the neuron is connected to.
When simulating neurons, the pulsation is ignored and the frequency of pulsation
modelled as a variable. Simulated neurons are typically called nodes. The axons
form the links in a directed graph connecting the nodes, and the directedness means
that nodes have input axons and output axons. Simulated axons are typically called
weights, largely because it is the value of the weight (representing the strength
of the connection) that is of primary interest. The weights of a neural network
are its parameters, and the job of the learning algorithm is to determine their
values. The qualitative description of the behaviour of neurons is of course given
a precise mathematical specification in simulated neural networks; this is provided
in Appendix 1 for the benefit of those who are interested.
A further simplification of the structure of the network is to arrange the nodes
into distinct layers. (It can be proved that this does not lead to loss of potential
functionality.) This simplification means that the choice of network structure is
simply a question of determining the number of layers, and for the layers that are
not input or output layers (the so-called hidden layers), the number of nodes to
use in each layer. The number of nodes in the input and output layers is of course
determined by the dimensionalities of the domain and range of the function to be
approximated. It has been proved (Cybenko 1989; Funahashi 1989; Hornik et al.
1989) that one hidden layer is sufficient to approximate any function. Although
having more hidden layers can mean that the contribution of the weights closer to