oliver.obst.eu

Information Processing in Echo State Networks at the Edge of Chaos

Joschka Boedecker, Oliver Obst, Joseph T. Lizier, Mayer, N. Michael, and Minoru Asada. Information Processing in Echo State Networks at the Edge of Chaos. In preparation., 2010.

Download

(unavailable)

Abstract

We investigate information processing in randomly connected recurrent neural networks. It has been shown previously that the computational capabilities of these networks are maximized when the recurrent layer is close to the border between a stable and an unstable dynamics regime, the so called edge of chaos. The reasons, however, for this maximized performance are not completely understood. We adopt an information-theoretical framework and are for the first time able to quantify the computational capabilities between elements of these networks directly as they undergo the phase transition to chaos. Specifically, we present evidence that both information transfer and storage in the recurrent layer peak close to this phase transition, providing an explanation for why guiding the recurrent layer towards the edge of chaos is computationally useful. As a consequence, our work suggests self-organized ways of improving performance in recurrent neural networks, driven by input data. Moreover, the networks we study share important features with biological systems such as feedback connections and online computation on input streams. A key example is the cerebral cortex, which was shown to also operate close to the edge of chaos. Consequently, the behavior of model systems as studied here is likely to shed light on reasons why biological systems are tuned into this specific regime.

BiBTeX Entry

@Article{	  BOLMA10,
  author	= {Joschka Boedecker and Oliver Obst and Joseph T. Lizier
		   and Mayer, N. Michael and Minoru Asada},
  journal	= {In preparation.},
  keywords	= {Recurrent Neural Networks, Reservoir Computing,
		   Information Transfer, Active Information Storage, Phase Transition},
  title 	= {Information Processing in Echo State Networks at the Edge
		   of Chaos},
  year		= {2010},
  abstract	= {We investigate information processing in randomly
		   connected recurrent neural networks. It has been shown previously that the
		   computational capabilities of these networks are maximized when the
		   recurrent layer is close to the border between a stable and an unstable
		   dynamics regime, the so called \emph{edge of chaos}. The reasons, however,
		   for this maximized performance are not completely understood. We adopt an
		   information-theoretical framework and are for the first time able to
		   quantify the computational capabilities between elements of these networks
		   directly as they undergo the phase transition to chaos. Specifically, we
		   present evidence that both information transfer and storage in the recurrent
		   layer peak close to this phase transition, providing an explanation for why
		   guiding the recurrent layer towards the edge of chaos is computationally
		   useful. As a consequence, our work suggests self-organized ways of improving
		   performance in recurrent neural networks, driven by input data. Moreover,
		   the networks we study share important features with biological systems such
		   as feedback connections and online computation on input streams. A key
		   example is the cerebral cortex, which was shown to also operate close to the
		   edge of chaos. Consequently, the behavior of model systems as studied here
		   is likely to shed light on reasons why biological systems are tuned into
		   this specific regime. }}