Improving Recurrent Neural Network Performance using Transfer Entropy September 15
Our new paper, Improving Recurrent Neural Network Performance using Transfer Entropy, has been accepted at the 17th International Conference on Neural Information Processing (ICONIP) in Sydney.
In this paper, we present an approach to improve the hidden layer of recurrent neural networks, guided by the learning goal of the system, and apply this new method to reservoir computing approaches. Reservoir computing uses, in general, a fixed, randomly initialised hidden layer. A consequence of this is that performance is usually quite good, but not necessarily optimal for the task at hand. There exist self-organised approaches – like intrinsic plasticity – that are able to improve performance of reservoir computing approaches, but usually, they just consider the input to the system, and don’t take the actual task of the system into account.
Our reservoir adaptation optimises the information transfer at each individual unit, dependent on properties of the information transfer between input and (desired) output of the system. Using synthetic data, we show that this reservoir adaptation improves the performance of offline echo state learning and Recursive Least Squares Online Learning.
ICONIP 2010 takes place from the 22nd–25th November 2010 in Sydney, Australia.
