oliver.obst.eu

Improving Recurrent Neural Network Performance using Transfer Entropy

Oliver Obst, Joschka Boedecker, and Minoru Asada. Improving Recurrent Neural Network Performance using Transfer Entropy. In Wong, Kok Wai, Mendis, B. Sumudu U., and Bouzerdoum, Abdesselam, editors, Neural Information Processing. Models and Applications, Lecture Notes in Computer Science, pp. 193–200, Springer, 2010.

Download

[PDF] (206 kb)

Abstract

Reservoir computing approaches have been successfully ap- plied to a variety of tasks. An inherent problem of these approaches, is, however, their variation in performance due to fixed random initialisa- tion of the reservoir. Self-organised approaches like intrinsic plasticity have been applied to improve reservoir quality, but do not take the task of the system into account. We present an approach to improve the hidden layer of recurrent neural networks, guided by the learning goal of the system. Our reservoir adaptation optimises the information transfer at each individual unit, dependent on properties of the information transfer between input and output of the system. Using synthetic data, we show that this reservoir adaptation improves the performance of offline echo state learning and Recursive Least Squares Online Learning.

BiBTeX Entry

@InCollection{	  OBA10,
  abstract	= {Reservoir computing approaches have been successfully ap-
		   plied to a variety of tasks. An inherent problem of these approaches, is,
		   however, their variation in performance due to fixed random initialisa- tion
		   of the reservoir. Self-organised approaches like intrinsic plasticity have
		   been applied to improve reservoir quality, but do not take the task of the
		   system into account. We present an approach to improve the hidden layer of
		   recurrent neural networks, guided by the learning goal of the system. Our
		   reservoir adaptation optimises the information transfer at each individual
		   unit, dependent on properties of the information transfer between input and
		   output of the system. Using synthetic data, we show that this reservoir
		   adaptation improves the performance of offline echo state learning and
		   Recursive Least Squares Online Learning. },
  author	= {Oliver Obst and Joschka Boedecker and Minoru Asada},
  booktitle	= {Neural Information Processing. Models and Applications},
  doi		= {http://dx.doi.org/10.1007/978-3-642-17534-3_24},
  editor	= {Wong, Kok Wai and Mendis, B. Sumudu U. and Bouzerdoum,
		   Abdesselam},
  keywords	= {Information Transfer, echo state networks, recurrent
		   neural networks},
  pages 	= {193--200},
  publisher	= {Springer},
  series	= {Lecture Notes in Computer Science},
  title 	= {Improving Recurrent Neural Network Performance using
		   Transfer Entropy},
  volume	= {6444},
  year		= {2010},
}