PREDICTABILITY MINIMIZATION

(

**Jürgen Schmidhuber
Department of Computer Science
University of Colorado
Campus Box 430,
Boulder, CO 80309, USA
yirgan@cs.colorado.edu**

I propose a novel general principle for unsupervised learning of distributed non-redundant internal representations of input patterns. The principle is based on two opposing forces. For each representational unit there is an adaptive predictor which tries to predict the unit from the remaining units. In turn, each unit tries to react to the environment such that it minimizes its predictability. This encourages each unit to filter `abstract concepts' out of the environmental input such that these concepts are statistically independent of those upon which the other units focus. I discuss various simple yet potentially powerful implementations of the principle which aim at finding binary factorial codes (Barlow et al., 1989), i.e. codes where the probability of the occurrence of a particular input is simply the product of the probabilities of the corresponding code symbols. Such codes are potentially relevant for (1) segmentation tasks, (2) speeding up supervised learning, (3) novelty detection. Methods for finding factorial codes automatically implement Occam's razor for finding codes using a minimal number of units. Unlike previous methods the novel principle has a potential for removing not only linear but also non-linear output redundancy. Illustrative experiments show that algorithms based on the principle of predictability minimization are practically feasible. The final part of this paper describes an entirely local algorithm that has a potential for learning unique representations of extended input sequences.

- INTRODUCTION
- FORMULATING THE PROBLEM
- THE BASIC PRINCIPLE AND ARCHITECTURE
- OBJECTIVE FUNCTIONS FOR THE THREE CRITERIA
- AN ERROR FUNCTION FOR THE INDEPENDENCE CRITERION
- AN OBJECTIVE FUNCTION FOR THE BINARY CRITERION
- AN ERROR FUNCTION FOR THE INVERTIBILITY CRITERION
- COMBINING ERROR TERMS
- REMOVING THE VARIANCE TERM: REAL-VALUED CODES
- REMOVING THE GLOBAL INVERTIBILITY TERM
- A DISADVANTAGE OF THE ABOVE METHODS

- LOCAL CONDITIONED VARIANCE MAXIMIZATION
- `NEURAL' IMPLEMENTATION
- EXPERIMENTS

- PREDICTABILITY MINIMIZATION AND TIME
- CONCLUDING REMARKS, OUTLOOK
- ACKNOWLEDGEMENTS
- Bibliography
- About this document ...

Back to Independent Component Analysis page.