oliver.obst.eu

(also posted on Google+)

I really like the summary of some of the recent approaches to recurrent neural network training in Yoshua Bengio et al.’s arxiv paper “Advances in Optimizing Recurrent Neural Networks [1]“. It’s also cool to see the effect of careful re-formulations, and some connections from deep learning to RNNs pointed out. From these areas I would expect to more exchange and connections down the track (for other papers to come). I don’t quite agree with the opener, “after a more than decade-long period of relatively little research activity in the area of recurrent neural networks …”, though. There has been quite some activity, and the Jaeger/Haas Science paper [2] (almost 10 years ago) on Echo State Networks was probably one of the starting points. My initial thought was that the training (or better the non-training) of the RNN hidden layer in reservoir methods may be the reason for the dismissal, but then, (more…)

In our new paper, we come up with a method that uses feedforward training for recurrent neural networks, in a somewhat unusual way. Our new approach “Tamed Reservoirs” results in better performance for time series prediction compared to Echo State Networks, and also reduces some of the variance in performance, since all of the reservoir connections are trained. Abstract and full PDF are available here.

We are organising a special session on Interactive Data Analysis and Visualization at the 2012 International Joint Conference on Neural Networks! The special session web site contains all the details.

By offering automated information extraction tools from data, machine learning has revolutionized the way in which humans can cope with electronic data volumes. The ever increasing complexity of the settings continues to pose challenges to the field: often, it is no longer possible to specify a priori a formal learning task; complex parameter choices can severely influence the outcome; and an appropriate encoding of data is not clear at all. More and more often, the human constitutes an important step in the loop to interactively decide about an appropriate learning model, model parameters, and data representation. Because of this fact, intuitive models and model parameters, and human understandable interfaces to the model and data are needed. In this frame, interesting new technologies have been developed such as high quality data visualization tools, sparse interpretable data representation and models, informed priors, active learning, and similar.

This special session aims to foster research in neural learning paradigms which offer an intuitive interface to data or models and thus have the potential as parts of an interactive pipeline. Go to the special session web site more information.

In our new paper (Relating Fisher information to order parameters. Physical Review E, 84(4):041116, 2011), we study phase transitions and relevant order parameters via statistical estimation theory using the Fisher information matrix. The assumptions that we make limit our analysis to order parameters representable as a negative derivative of thermodynamic potential over some thermodynamic variable. Nevertheless, the resulting representation is sufficiently general and explicitly relates elements of the Fisher information matrix to the rate of change in the corresponding order parameters. The obtained relationships allow us to identify, in particular, second-order phase transitions via divergences of individual elements of the Fisher information matrix. A computational study of random Boolean networks (RBNs) supports the derived relationships, illustrating that Fisher information of the magnetization bias (that is, activity level) is peaked in finite-size networks at the critical points, and the maxima increase with the network size. The framework presented here reveals the basic thermodynamic reasons behind similar empirical observations reported previously. The study highlights the generality of Fisher information as a measure that can be applied to a broad range of systems, particularly those where the determination of order parameters is cumbersome.

We are organising a special session “Information Processing, Inference and Learning” at the Fifth Australian Conference on Artificial Life (ACAL11).

Organisers: Oliver Obst, Mikhail Prokopenko (CSIRO ICT Centre)

Keynote speaker (tentative): Prof. Martin Riedmiller, Albert-Ludwigs University Freiburg, Germany

Learning is one of the most important capabilities of living systems, and processes involved in inference and learning are ingrained in many levels of organisation scale, e.g., from single neurons to organisations or societies. Much of the information processing involved in these processes is consequently distributed, and relies on local interactions among individual entities. On the other hand, many successful approaches from machine learning require some centralised processing, while in living systems no such requirement exists.

In this special session, we are looking for contributions that will address issues that involve information processing, inference, and learning, with a perspective on their application to Alife scenarios. These include for example methods to help characterising relevant information, self-organised encoding of information, decentralised approaches to learning, and inference mechanisms. Possible topics include, but are not limited to,

• information theoretic methods for analysing/describing Alife scenarios
• interactions between genetic evolution, and individual learning
• self-organised information processing
• intrinsically motivated learning
• guided self-organisation

To submit papers to this special session, please use the ACAL 2011 submission system and use Information Processing, Inference and / or Learning as a keyword.

(more…)

Our new paper Phase transitions in least-effort communications has been published in J. Stat. Mech. In this work, we critically examine a model that attempts to explain the emergence of power laws (e.g., Zipf’s law) in human language. The model is based on the principle of least effort in communications—specifically, the overall effort is balanced between the speaker effort and listener effort, with some trade-off. It has been shown that an information-theoretic interpretation of this principle is sufficiently rich to explain the emergence of Zipf’s law in the vicinity of the transition between referentially useless systems (one signal for all referable objects) and indexical reference systems (one signal per object). The phase transition is defined in the space of communication accuracy (information content) expressed in terms of the trade-off parameter. Our study (more…)

DMMD 2011 Symposium: Distributed machine learning and sparse representation with massive data sets
Web page: http://research.ict.csiro.au/conferences/machine-learning/

The symposium will take place at the CSIRO Campus in Sydney (Marsfield), Australia.

The exponentially increasing demand for computing power as well as physical and economic limitations has contributed to a proliferation of distributed and parallel computer architectures. To make better use of current and future high-performance computing, and to fully benefit from these massive amounts of data, we must discover, understand and exploit the available parallelism in machine learning. Simultaneously, we have to model data in an adequate manner while keeping the models as simple as possible, by making use of a sparse representation of the data or sparse modelling of the respective underlying problem.

The invited speakers are:

Samy Bengio (Google Research, CA, USA)
Barbara Hammer (University of Bielefeld, Germany)
Yann LeCun (New York University, NY, USA)
Michael Mahoney (Stanford University, CA, USA)

Call for Papers / Extended Abstracts

(more…)

Our new paper, Improving Recurrent Neural Network Performance using Transfer Entropy, has been accepted at the 17th International Conference on Neural Information Processing (ICONIP) in Sydney.

In this paper, we present an approach to improve the hidden layer of recurrent neural networks, guided by the learning goal of the system, and apply this new method to reservoir computing approaches. Reservoir computing uses, in general, a fixed, randomly initialised hidden layer. A consequence of this is that performance is usually quite good, but not necessarily optimal for the task at hand. There exist self-organised approaches – like intrinsic plasticity – that are able to improve performance of reservoir computing approaches, but usually, they just consider the input to the system, and don’t take the actual task of the system into account.

Our reservoir adaptation optimises the information transfer at each individual unit, dependent on properties of the information transfer between input and (desired) output of the system. Using synthetic data, we show that this reservoir adaptation improves the performance of offline echo state learning and Recursive Least Squares Online Learning.

ICONIP 2010 takes place from the 22nd–25th November 2010 in Sydney, Australia.

The Third International Workshop on Guided Self-Organisation (GSO-2010) will be held at Indiana University in Bloomington, Indiana, USA, 4-6 September 2010.

The workshop is comprised of a group of researchers with diverse yet related interests, overlapping in the area of self-organizing systems and methods for characterizing those systems in ways that may ultimately allow them to be guided toward prespecified goals. Information theory and graph theory are core to many of these methods; quantifying complexity and its sources a common theme.

If interested in participating, send an extended abstract to the email addresses on the workshop web site.  Selected works from the workshop will likely be published in a special journal issue (as has been the case in the past).  More information on the GSO-2010 web site.

guided self-organisation, workshop, artificial life, ai

You are invited to submit to and/or attend The First Australasian Workshop on Computation in Cyber-Physical Systems (CompCPS-2010).
We are organising this event here in Sydney, on the 15-16 July, in the Lecture Theatre at the CSIRO Marsfield site.

The name “cyber-physical system” (CPS) was chosen by the NSF and other United States federal agencies for systems that coherently combine computational and physical elements.

The CPS field builds up on knowledge and practical experiences of embedded systems, sensor networks, multi-robot teams, modular/swarm robotics, amorphous computing, programmable materials, evolvable/adaptive hardware, etc., and yet promise to form a unique field.

This Workshop will focus on distributed computation in CPS – the computation processes that integrate multiple data streams, compress and structure high-dimensional information, synchronise the distributed dynamics, adapt to topological changes within networks, optimise multiple sensorimotor loops, etc.

Several prominent invited speakers from Australia, Spain and USA will present different aspects of this rapidly developing research field.

Anyone interested in participating in the workshop is encouraged to submit a two-page extended abstract by May 16, 2010. Notifications will be sent by June 11, 2010 to all those who will be invited to the workshop. All accepted submissions will be allocated an oral presentation slot. See the Workshop Web Page for details.

cyber-physical systems, workshop, australia, call for papers, Sydney, australasia, sensor networks