oliver.obst.eu

Initialization and Self-Organized Optimization of Recurrent Neural Network Connectivity

Joschka Boedecker, Oliver Obst, Norbert Michael Mayer, and Minoru Asada. Initialization and Self-Organized Optimization of Recurrent Neural Network Connectivity. HFSP Journal, 3(5):340–349, October 2009.

Download

[PDF] (387 kb) [web page]

Abstract

Reservoir computing (RC) is a recent paradigm in the field of recurrent neural networks. Networks in RC have a sparsely and randomly connected fixed hidden layer, and only output connections are trained. RC Networks have recently received increased attention as a mathematical model for generic neural microcircuits, to investigate and explain computations in neocortical columns. Applied to specific tasks, their fixed random connectivity, however, leads to significant variation in performance. Few problem specific optimization procedures are known, which would be important for engineering applications, but also in order to understand how networks in biology are shaped to be optimally adapted to requirements of their environment. We study a general network initialization method using permutation matrices and derive a new unsupervised learning rule based on intrinsic plasticity (IP). The IP based learning uses only local learning, and its aim is to improve network performance in a self-organized way. Using three different benchmarks, we show that networks with permutation matrices for the reservoir connectivity have much more persistent memory than the other methods, but are also able to perform highly non-linear mappings. We also show that IP based on sigmoid transfer functions is limited concerning the output distributions that can be achieved.

BiBTeX Entry

@Article{	  BOMA09b,
  author	= {Joschka Boedecker and Oliver Obst and Norbert Michael
		   Mayer and Minoru Asada},
  doi		= {10.2976/1.3240502},
  journal	= {HFSP Journal},
  keywords	= {recurrent neural networks, intrinsic plasticity,
		   reservoir computing},
  month 	= oct,
  number	= {5},
  pages 	= {340--349},
  title 	= {Initialization and Self-Organized Optimization of
		   Recurrent Neural Network Connectivity},
  url		= {http://dx.doi.org/10.2976/1.3240502},
  volume	= {3},
  year		= {2009},
  abstract	= {Reservoir computing (RC) is a recent paradigm in the
		   field of recurrent neural networks. Networks in RC have a sparsely and
		   randomly connected fixed hidden layer, and only output connections are
		   trained. RC Networks have recently received increased attention as a
		   mathematical model for generic neural microcircuits, to investigate and
		   explain computations in neocortical columns. Applied to specific tasks,
		   their fixed random connectivity, however, leads to significant variation in
		   performance. Few problem specific optimization procedures are known, which
		   would be important for engineering applications, but also in order to
		   understand how networks in biology are shaped to be optimally adapted to
		   requirements of their environment. We study a general network initialization
		   method using permutation matrices and derive a new unsupervised learning
		   rule based on intrinsic plasticity (IP). The IP based learning uses only
		   local learning, and its aim is to improve network performance in a
		   self-organized way. Using three different benchmarks, we show that networks
		   with permutation matrices for the reservoir connectivity have much more
		   persistent memory than the other methods, but are also able to perform
		   highly non-linear mappings. We also show that IP based on sigmoid transfer
		   functions is limited concerning the output distributions that can be
		   achieved. },
}