Biological Foundations
of Neural Nets and simple artificial modelsBiological Foundations
of Neural Nets and simple artificial modelsMost artificial neural networks are based to some degree on biological systems. The function of the Processing Elements is based loosely on the nerve cell.
If the total input potential is sufficient then the neuron fires. An action potential is generated and propagates down the axon towards the synaptic junctions with other nerve cells.
From an Engineering perspective:
Neural information is frequency encoded.
McCullock and Pitts suggested the first synthetic neuron in the early 1940s. In the McCullock-Pitts model the artificial neuron produces a binary output whose value depends on the weighted sum of its inputs.
The information processing performed in this way may be crudely summarised as follows: signals (action-potentials) appear at the unit's inputs (synapses). The effect (PSP) each signal has may be approximated by multiplying the signal by some number or weight to indicate the strength of the synapse. The weighted signals are now summed to produce an overall unit activation. If this activation exceeds a certain threshold the unit produces a an output response. This functionality is captured in the artificial neuron known as the Threshold Logic Unit (TLU) originally proposed by McCulloch and Pitts.
We suppose there are n inputs with signals x1, x2, ... xn and weights w1, w2, ... wn. The signals take on the values '1' or '0' only. That is the signals are Boolean valued. (This allows their relation to digital logic circuits to be discussed). The activation a, is given by
a = w1x1 + w2x2 + ... wnxn
This may be represented more compactly as
the output y is then given by thresholding the activation
y = {
1 if a >= ß
0 if a < ß
The threshold ß will often be zero. The threshold function is sometimes called a step-function or hard-limiter for obvious reasons. If we are to push the analogy with real neurons, the presence of an action-potential is denoted by binary `1' and its absence by binary `0'.
In artificial neural networks, different activation functions are used. NNs with the identity function only support linear models. The sigmoid function lets you model higher order functions. For an example of how this works, see a JavaScript model of a neuron in the diggers network example from Neural Planner.
When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased (Hebb 1949)
In an artificial neural network Hebbian learning is associated with varying the weighting factors, Wi, between neurons.
The process of adjusting the weights is referred to as a Learning Algorithm.
There are two main phases in the operation of a network:
During learning the connection weights are changed in response to stimuli presented at the input buffer and, optionally, the output buffer.
Learning may be:
You can step through an example of supervised learning in a perceptron (an early, simple, NN model), run Fred Corbett's Perceptron Learning Applet (needs Java), see a graphical visualisation of a self-organising Kohonen feature map from Jochen Fröhlich's thesis, or train and test a handwriting recogniser.
Recall refers to how the network processes a stimulus and produces a response.
The input may have been in the original training set or may have been previously unseen.
