oliver.obst.eu

Our new paper, Improving Recurrent Neural Network Performance using Transfer Entropy, has been accepted at the 17th International Conference on Neural Information Processing (ICONIP) in Sydney.

In this paper, we present an approach to improve the hidden layer of recurrent neural networks, guided by the learning goal of the system, and apply this new method to reservoir computing approaches. Reservoir computing uses, in general, a fixed, randomly initialised hidden layer. A consequence of this is that performance is usually quite good, but not necessarily optimal for the task at hand. There exist self-organised approaches – like intrinsic plasticity – that are able to improve performance of reservoir computing approaches, but usually, they just consider the input to the system, and don’t take the actual task of the system into account.

Our reservoir adaptation optimises the information transfer at each individual unit, dependent on properties of the information transfer between input and (desired) output of the system. Using synthetic data, we show that this reservoir adaptation improves the performance of offline echo state learning and Recursive Least Squares Online Learning.

ICONIP 2010 takes place from the 22nd–25th November 2010 in Sydney, Australia.

Our new paper describes a mathematical model for generic neural microcircuits, with potential engineering applications, as well as implications to understand how networks in biology are shaped to be optimally adapted to requirements of their environment.

Reservoir computing (RC) is a recent paradigm in the field of recurrent neural networks. Networks in RC have a sparsely and randomly connected fixed hidden layer, and only output connections are trained. RC networks have recently received increased attention as a mathematical model for generic neural microcircuits to investigate and explain computations in neocortical columns. Applied to specific tasks, their fixed random connectivity, however, leads to significant variation in performance. Few problem-specific optimization procedures are known, which would be important for engineering applications, but also in order to understand how networks in biology are shaped to be optimally adapted to requirements of their environment. We study a general network initialization method using permutation matrices and derive a new unsupervised learning rule based on intrinsic plasticity (IP). The IP-based learning uses only local learning, and its aim is to improve network performance in a self-organized way. Using three different benchmarks, we show that networks with permutation matrices for the reservoir connectivity have much more persistent memory than the other methods but are also able to perform highly nonlinear mappings. We also show that IP-based on sigmoid transfer functions is limited concerning the output distributions that can be achieved.

neurophysiology, optimisation, physiological models, recurrent neural nets, unsupervised learning, reservoir computing, echo state networks

In a paper that was recently accepted at the European Symposium on Artificial Neural Networks (ESANN 2009), we look at different ways to influence the performance of echo state networks. Traditionally, echo state networks and other reservoir computing approaches use a fixed random connected reservoir, which leads to significant variation in performance. Only few problem specific optimisation procedures are known to date. We study a general initialization method using permutation matrices and derive a new unsupervised learning rule based on intrinsic plasticity (IP) for echo state networks. Using three different benchmarks, we show that networks with permutation matrices for the reservoir connectivity have much longer memory than the other methods, but are also able to perform highly non-linear mappings. We also show that IP based on sigmoid transfer functions is limited concerning the output distributions that can be achieved.

Studies on Reservoir Initialization and Dynamics Shaping in Echo State Networks,
J. Boedecker, O. Obst, N.M. Mayer, M. Asada. The full paper will be available after the conference (April) is now available.

neural networks, intrinsic plasticity, echo state networks, reservoir computing