Neural Networks | Reinforcement Learning | Evolutionary Algorithms

My current research focuses on artificial neural networks, recurrent neural networks, evolutionary algorithms and deep-learning applied to reinforcement learning, control problems, image classification, handwriting and speech recognition.

At IDSIA, I have been working on EU funded projects Humanobs and ``Nascence'', SNF project ``Theory and Practice in Reinforcement Learning 2'' as well as projects funded from industrial cooperation.

In state-of-the-art neuroevolution, researchers look for efficient way of encoding the artificial neural networks in strings (genomes) of symbols (genes) in order to reduce the search space of such genomes. Jan's recent research in indirect encoding lead in a method, which describes a neural network weight matrix by a limited set of it's frequency coefficient. The genome consists of a limited set of frequency coefficients that transform to the weight matrix using inverse Fourier-type frequency transform.

The weight matrix get decorrelated after transformed to the frequency domain. The complexity of a genome could be pushed down by encoding the frequency coefficients with a limited number of bits. If the space of coefficients is small (say less than 32 bits), then it could be searched exhaustively starting from the shortest bit strings.

Surprisingly, some of the well known benchmarks could be solved with networks described by fairly short bit-string. For example, single-pole balancing controller consisting of one neuron could be described by just 1 bit (positive constant weights matrix), which means that single-pole benchmark no longer exists.

Manipulation with a flexible octopus arm [Yekutieli et al.] with 10 compartments is a control problem with 82 continuous state variables and 32 control variables. A fully-connected RNN with 3680 weights, can be evolved in 20-dimensional coefficient space in order to learn to reach a target [12d].

So how to do it? Suppose that at every fitness evaluation a genotype is transformed to a phenotype by simply placing the genomes to a weight matrix (or a vector):

The easiest usage of compressed NN is to include the code that takes the genes, pafs them with zeroes and performs inverse DCT. A simple Mathematica code for doing this would then be:

The compressed encoding was tested on more challenging RL problem that involve processing of raw vision observations stream. The scaled up RNN controllers were evaluated on two vision-based benchmarks: the visual octopus arm, in which the controller has no access to the arm state variables but is input a raw image stream that contains the arm visualization.

In the second case, it controls a race car to drive along a track using a visual stream captured from the driver's perspective. Such a network has over 1.1 million weights To our knowledge this the first attempt to tackle TORCS using vision, and successfully evolve neural network controllers of this size [13c].

A Clockwork RNN [14c], a simple yet powerful modification to the simple RNN (SRN), in which the hidden layer is partitioned into separate modules, each processing inputs at its own temporal granularity, making computations only at its prescribed clock rate. It reduces the number of SRN parameters, improves the performance significantly in the sequential tasks, and speeds up the network evaluation. It has been shown experimentally, that it outperforms LSTM [Hochreiter-1997] in audio signal generation and TIMIT spoken word classification.

The goal of sequence generation task is to train a recurrent neural network, that receives no input, to generate a target sequence as accurately as possible. The weights of the network can be seen as a (lossy) encoding of the whole sequence, which could be used for compression. SRN tends to learn the first few steps of the sequence and then generates the mean of the remaining portion, while the output of LSTM resembles a sliding average, and CW-RNN approximates the sequence much more accurately.

Temporal Hebbian Self-organizing Map [7b,8a] is a recurrent extension of Kohonen's SOM. Additional layer of full recurrent connections among the nodes is trained in a Hebbian way. The connections accumulate first-order statistics of transitions between states represented by the neurons, while placing the neurons into centroids of clusters using the input connections. The network clusters the data in both input space and time. The initial THSOM model Hebbian training was improved by [Ferro et al.] introducing neighborhood in the temporal connections (Topological THSOM). The network was further extended for unsupervised learning of a model in RL and named Temporal Network for Transitions (TNT) [11a].

2016 IFI Summer School, slides, gradient example notebook