Neural Networks of Dendritic Networks?

VOLUME 4, FEBRUARY 2011

Neural Networks of Dendritic Networks?

By : Khalid Isa

The above question was given by Dr. Loo Chu Kiong from Multimedia University during the Symposium on Autonomous Vehicle Development in Malaysia on January 13, 2011. He said that the neural networks approach which currently being used as robot brain only focus on the surface of human brain neurons. Due to that, the ability of autonomous robot to be intelligent as human brain is far from perfection. In order to overcome the weakness of neural networks, researchers should explore the dendrites instead of neurons, because the number of dendrites inside human brain is more than the number of neurons. But the real question is what is dendrites and how to produce the dendritic computations?

 

feb 1a

Figure 1 Neuron's dendritic tree

Dendrites are short, thick branched extensions which extend like the roots of a tree over other neurons or body cells. The dendrites all branch off dendritic spines, which in turn branch of the cell body. Dendrites are the receptive sites of the neurons. Here, the neurons receive electric messages from other neurons or body cells. The site where one dendrite meets another neuron's impulse is called the synapse. Usually, neurons have hundreds of dendrite extensions. These extensions are spread over a large area, giving the neuron better reception of signals. Some dendrites are specialized for the accumulation of information. These cells are finer than other dendrites and found near the brain.

Based on the literature, the computations of dendritic networks can be divided into two: passive dendritic and active dendritic. The first thing that we need to understand is the computations in passive dendritic. It is important to recognize that the passive properties of the dendritic tree provide the backbone for the electrical signaling in dendrites, even when they are richly endowed with voltage-dependent ionic currents. For example, the threshold for initiation of a dendritic spike is determined in part by the availability of sodium channels, but perhaps even more by the passive load of the surrounding dendrites, which dictates how much of the input current will depolarize the membrane and how much will flow axially toward other dendritic regions [1]. In terms of signal propagation, dendrites behave like electrical cables with medium-quality insulation. As such, passive dendrites linearly filter the input signal as it spread to the site of initiation, where its compared with the threshold. This filtering tends to attenuate the dendritic signal as a function of the distance it travels and the frequency of the original signal.

The understanding of computations in active dendritic also crucial. The active properties of the dendritic tree provide a feedback mechanism. Solely on the basis of anatomical observations states that in the nervous system information flows in one direction: from dendrites to soma to axon. In the past decade it has become clear that in many types of neurons the presence of excitable ionic currents in the dendrites supports dendritic action potentials that travel in the reverse direction, from the soma into the dendrites [2]. Computationally this “backpropagation” has major consequences because it implies that the neuron is no longer an open-loop system but has an internal mechanism of feedback. It is thus no longer the case that feedback is a property only of the network, but rather it is a salient property of each element of the network. Moreover, the feedback conveyed by the backpropagating action potential is highly sophisticated and has many important consequences for dendritic function, and also for synaptic plasticity [3].

Although dendrites have been studied for decades, the field of dendritic computation is still in its infancy. This is partly because dendrites remain relatively inaccessible and have only recently begun to yield their secrets to the onslaught of multiple new experimental tools. However, the real challenge is deeper one: how to evaluate the importance of mechanisms on the molecular and cellular level for computation at the behavior level. The ability not only to record electrical and chemical signals in the intact brain but also to manipulate the structure and function of dendrites using molecular tools will hopefully allow us to move from the descriptive level, correlating dendritic signals linked to computation with behavior, toward directly testing the causal nature of these links. Such experiments will provide a deeper understanding of how single neurons contribute to computation in the brain and should inspire the development of novel neural network architectures with the computational powers of real brains.

References

[1] Segev I, London M., 1999, A theoretical view of passive and active dendrites, In Dendrites, ed. G Stuart, N Spruston, M Hausser, pp. 205–30. Oxford, UK: Oxford Univ. Press.

[2] Stuart G, Schiller J, Sakmann B., 1997, Action potential initiation and propagation in rat neocortical pyramidal neurons, J. Physiol, 505:617–32.

[3] Linden D., 1999, The return of the spike: postsynaptic action potentials and the induction of LTP and LTD. Neuron 22:661–66.

[4] London, M., Hausser, M., 2005, Dendritic Computation, Annual Revision Neuroscience, pp. 503-532.

Modern Control Systems_continue from previous issue

VOLUME 4, JANUARY 2011

Modern Control Systems_continue from previous issue

By : Maziyah Mat Noh

System Analysis

There are two important properties of linear systems that determine whether or not given control objectives are achievable.

1. A system is said to be controllable if it is possible to find a control input that takes the system from any initial state to any final state in any given time interval

2. A system is said to be observable if the value of the state variables can be uniquely determined from its input and output signals

Controllability

Consider the following state space model

jan 4a

The system (1) is controllable if and only if the controllability matrix C(A,B) has full rank (rank = n). When controllability matrix is full rank then the controllability gramian, jan 4b is positive definite for all t > 0

jan 4c jan 4d

For the above example for spring-mass-damper system we have the state space equation as follows:

jan 4e jan 4f

if we choose the spring constant,k = 1, mass, m = 0.1kg, and damping coefficient, b = 0.05. The matlab simulation the give controllability

and has a rank = 2 (which is full rank). Therefore the above system is controllable. The eigenvalue of the system is -0.2500 ± 3.1524i. Meaning the poles of the system are lies on left-hand-plane.

Observability

The system (1) is observable if and only if the observability matrix O(C,A) has full rank (rank = n). When observability matrix is full rank then the observability gramian, is positive definite for all t > 0. With same example, we obtain the observability matrix, O(C,A) is given by jan 4h and has full rank, i.e rank = 2.

jan 4i [6] jan 4j

Therefore this system, the system is controllable and observable. With these tests we then can proceed to design an appropriate controller for the system.

References

1. Ashish Tewari, Modern Control Design with MATLAB and SIMULINK, John Wiley & Sons Ltd, 2002

2. Wikipedia. Control Theory .http://en.wikipedia.org/wiki/Control_theory. Accessed November 2010.

Principles of Underwater Acoustic Transducer Design (Part 4)

VOLUME 4, JANUARY 2011

Principles of Underwater Acoustic Transducer Design (Part 4)

By : Mohd Ikhwan Hadi Yaacob

Projector Design Projector plays very important role in active sonar system. It ensures the detection of the transmitted sound by the hydrophone from a target. The fundamental behind projector design remain the same with hydrophone, however transmitting high intensity sound requires complex design than a hydrophone. For certain applications such as short distance obstacle avoidance [1] and low range high resolution imaging for compact ROV/AUV, projector and hydrophone within same transducer module (transceiver) is nearly made possible through microelectro-mechanical system and miniaturization technologies. High power underwater sound projectors have proven significant roles in naval applications for active-search sonar and mine-hunting sonar. Medium power projectors on the other hand can be found in depth sounder, communication and submarine obstacle avoidance. For port surveillance, projector design has become extremely challenging in order to transmit extra high power-low frequency signal for long-range active surveillance.As discussed briefly in previous issue, projectors can be found in various design namely spheres, ring, piston or flextensional; depending on the system requirements.

For several reasons, piezoelectrics are the most commonly utilized in projector: easy fabrication, low electrical losses and strong mechanical properties. However, projector performance (power intensity, bandwidth and efficiency) is limited by its size and weight. Recently, various research works has focused on piezoelectric micromachined ultrasonic transducer (pMUT) for underwater sound projector producing micro scale transducer array with higher transmit efficiency at lower power consumptions. It was predicted that miniaturized underwater acoustic transducer will be common for various marine applications for the next coming years [2]. Table 1 lists important score for each type of conventional underwater projectors [3]:

jan 3a

Table 1: Type of projectors

 

Two figures of merit (FOM) typically being used to represent projectors which are total transducer volume (FOMv) and total transducer mass (FOMm). Let say the volume of active material is given by AL, with A is area and L is length, FOMv which is V0 > AL can be written as; FOMv = W / (Vo fr Qm) where W is total power, fr is resonance frequency and Qm is mechanical Q. FOMv is calculated in Watts/Hz m3. When weight of projector is more important than size, total transducer mass FOMm can be used which is W/ (M fr Qm), with M is mass. FOMm is determine in the unit of Watts/kHz kg. various FOM of piezoelectric flextensional transducer [3] is listed in Figure 2.

jan 3b

Table 2: FOM of piezoelectric flextensional transducer

 

At a specific given frequency, projector which produce greater power within a smaller volume for the least weight and lower Qm will have higher FOM. These numbers provide direct comparison between projectors within the same operating conditions. At this point, MEMS based acoustic projector offers far better potential to carry higher FOM owing to its small volume, less weight, low Qm while deliberately deliver enough power outputs [4].

References

[1] H. Lee, D. Kang and W. Moon, “Micro electro-mechanical based parametric transmitting array in air – application to the ultrasonic ranging transducer with high directionality,” IEEE Proceedings on SICE-ICASE International Joint Conference, pp. 1081-1084, 2006.

[2] M.R. Arshad, Recent advancement in sensor technology for underwater applications. Indian J. Mar. Sci., 38(3), 267-273 (2009).

[3] C.H. Sherman and J.L. Butler, Transducers and Arrays for Underwater Sound, Springer 2007

 

Code Implementation _ continue from previous issue

VOLUME 4, JANUARY 2011

Code Implementation _ continue from previous issue

By: Zulkifli Zainal Abidin

 

1.2 The Smelling Algorithm

After a series of controlled randomizations, the flies will focus on the known best point. The surrounding of the known best points will be explored. This is the basic idea of the smelling algorithm. However, a form of direct application as such yields poor results. Firstly, the radius must not be fixed all of the time. A shrinking process must be properly applied onto the radius. It is found that the smelling radius shrinks a little at the beginning while critical drops to a small radius. Due to the fact that the accuracy is set to a certain resolution, the smelling radius will fall to a constant and not to zero. Thus, exponential approximation could be made but it must be expressed in a piece-wise manner. The first part illustrates the falling value of the radius at the onset to half of its initialized value. The second part illustrates the falling value of the radius from half the initialized value to its minimum.

jan 2a

 

Where D_1 and D_2 are shaping factors. The larger the shaping factors, the slower the converging of the exponential term. Decreasing smelling radius over a number of iterations is illustrated below. Secondly, the radius must bounce within a predetermined range. It is found that a bounce of 30% above and below the smelling radius yields better result. This will eliminate the effect of faster or slower shrinking rate of smelling radius. Thirdly, an offset angle must be accounted. If there is no random offset angle, only certain angles will be examined, and this will reduce the converging rate. However, the recommended offset angle is within the angles in between the flies set earlier. Thus, a general equation for smelling process would be:

Where D_1 and D_2 are shaping factors. The larger the shaping factors, the slower the converging of the exponential term. Decreasing smelling radius over a number of iterations is illustrated below. Secondly, the radius must bounce within a predetermined range. It is found that a bounce of 30% above and below the smelling radius yields better result. This will eliminate the effect of faster or slower shrinking rate of smelling radius. Thirdly, an offset angle must be accounted. If there is no random offset angle, only certain angles will be examined, and this will reduce the converging rate. However, the recommended offset angle is within the angles in between the flies set earlier.


Thus, a general equation for smelling process would be:

jan 2b

jan 2c

Where n is the number of flies that are involved in the smelling process, k is the number of flies among the involved flies while A and B are constants. A good estimation for A and B would be A+B/2=1 with both A and B around 0.6.

1.3 Tracking (Shooting) Process

After selecting the best direction, tracking process can then be conducted. Instead of having a varying angle as in the smelling process, the tracking process will instead have its radius varying for the flies involved. For this process, the number of the flies should be more than 16. Among these flies, the best fly will be chosen. The radius of the chosen fly is known as hit radius.

jan 2d

 

Figure 7 below illustrates the fact that shooting at the exact direction misses a chance to get nearer to the global best. Like the smelling radius, the maximum tracking radius and the minimum tracking radius shrinks over iterations. Logarithm scale is a good choice for the distance between tracking flies since a large leaping is unlikely to happen if compared to the points near to the known best point. It must be realized that known best only updates during the tracking process. Hence, the center will remain at the same point as long as no better solution is available. Defining UL to be the initial minimum radius, LL as the final minimum radius, UU as the initial maximum radius while LU is the final maximum radius, length is the number of iterations for the radius to settle down to the final radius, i is the number of current iterations, u and l are the current maximum radius and minimum radius respectively,

jan 2e

Due to the variation during the initialization process, the shrinking should be done at variable rate. Hence, if the hit radius decreases, the shrinking should be quickened. The reverse should also be true. However, shrinking must be controlled. Therefore, a shrinking scale is introduced. The shrinking scale is reduced (normally 0.975 of its original value) only when the hit radius is smaller than a very small threshold at 3 times the resolution. If it is not less than the threshold, the scale should increase at a very small rate (normally at 1.0005) so that the radius can be maintained. However, tracking in exact directions might end up giving poorer results. This is because the gradient is not always constant.

Thus, randomization in terms of angles must be made. The range of angles should be 20-40 degrees at each side.



jan 2f

Figure 8 above illustrates shooting at a suitable range of C will enhance the chances to get nearer to the global best point. Most problem domains do not exhibit a constant gradient. Thus, randomization in shooting angles will certainly help in terms of converging speeds. Undoubtedly, there would be also being chances of obtaining poorer results. However, the probability for such a thing to happen declines when the number of flies involved in the tracking increases. The position of fly number k is:

jan 2g

Where posx and posy are the coordinates of center, theta is the direction obtained from the smelling process while C is the angle bound in radian.

1.4 Randomization in the Midst of Smelling and Tracking

The purpose of randomization is to prevent the best known point to be trapped at the local minimum. In fact, the frequency of randomization is not as rapid as during smelling and tracking. Randomization should be started after the first 30-50 iterations and conducted at every 5 to 8 iterations after the first 30-50 iterations. The frequency of randomization should be increased when a number of local optimum appears. Since the main objective is to find the global optimum and then venture beyond the best known area, the alpha of Levy motion can now be decreased.

Artificial Neural Networks

VOLUME 4, JANUARY 2011

Artificial Neural Networks

By: Khalid Bin Isa

An artificial neural network (ANNs) are a form of computation inspired by the structure and function of the natural neurons The neurons can be 'on' or 'off' and time is discrete. These natural neurons receive signals through synapses located on the dendrites or membrane of the neuron. When the signals received are strong enough (surpass a certain threshold), the neuron is activated and emits a signal though the axon. This signal might be sent to another synapse, and might activate other neurons.

 

jan 1kjan 1l

 

The complexity of real neurons is highly abstracted when modelling artificial neurons. These basically consist of inputs (like synapses), which are multiplied by weights (strength of the respective signals), and then computed by a mathematical function which determines the activation of the neuron. Another function (which may be the identity) computes the output of the artificial neuron (sometimes in dependence of a certain threshold). ANNs combine artificial neurons in order to process information. In weighting factors, the values jan 1a are weight factors associated with each node to determine the strength of input row vector jan 1b. Each input is multiplied by the associated weight of the neuron connection XTW. Depending upon the activation function, if the weight is positive, jan 1c commonly excites the node output; whereas, for negative weights, jan 1c tends to inhibit the node output. The higher a weight of an artificial neuron is, the stronger the input which is multiplied by it will be. Depending on the weights, the computation of the neuron will be different. By adjusting the weights of an artificial neuron we can obtain the output we want for specific inputs. But when we have an ANN of hundreds or thousands of neurons, it would be quite complicated to find by hand all the necessary weights. But we can find algorithms which can adjust the weights of the ANN in order to obtain the desired output from the network. This process of adjusting the weights is called learning or training. The node’s internal threshold jan 1d is the magnitude offset that affects the activation of the node output y as follows:

jan 1e

An activation function performs a mathematical operation on the signal output. More sophisticated activation functions can also be utilized depending upon the type of problem to be solved by the network. Five of the most common activation functions as described herein are also supported by MATLAB package.

a) Linear Function

As is known, a linear function satisfies the superposition concept. The mathematical equation for the above linear function can be written as

jan 1f

where a is the slope of the linear function. If the slope a = 1, then the linear activation function is called the identity function. The output (y) of identity function is equal to input function (u). Although this function might appear to be a trivial case, nevertheless it is very useful in some cases such as the last stage of a multilayer neural network.

b) Threshold Function

A threshold (hard-limiter) activation function is either a binary type or a bipolar type respectively. The output of a binary threshold function can be written as:

jan 1g

The neuron with the hard limiter activation function is referred to as the McCulloch-Pitts model.

c) Piecewise Linear Function

This type of activation function is also referred to as saturating linear function and can have either a binary or bipolar range for the saturation limits of the output. The mathematical model for a symmetric saturation function is described as follows:

jan 1h

d) Sigmoidal (S Shaped) Function

This nonlinear function is the most common type of the activation used to construct the neural networks. It is mathematically well behaved, differentiable and strictly increasing function. A sigmoidal transfer function can be written in the following form:

jan 1i

where a is the shape parameter of the sigmoid function. By varying this parameter, different shapes of the function can be obtained. This function is continuous and differentiable.

e) Tangent Hyperbolic Function

This transfer function is described by the following mathematical form:

jan 1j

References:

1. Fausett, L., Fundamentals of Neural Networks, Prentice-Hall, Englewood Cliffs, NJ, 1994.

2. Haykin, S., Neural Networks: A Comprehensive Foundation, Prentice Hall, Upper Saddle River, NJ, 1999.

3. Ham, F. and Kostanic, I., Principles of Neurocomputing for Science and Engineering, McGraw Hill, New York, NY, 2001.