Sepp Hochreiter, TUM
- Jürgen Schmidhuber, IDSIA
In M. S. Kearns, S. A. Solla, D. A. Cohn, eds., Advances in Neural Information Processing Systems 11, NIPS'11, p. 459-465, MIT Press, Cambridge MA, 1999.
This paper reveals a previously ignored connection between two important fields: regularization and independent component analysis (ICA). We show that at least one representative of a broad class of algorithms (regularizers that reduce network complexity) extracts independent features as a by-product. This algorithm is Flat Minimum Search (FMS), a recent general method for finding low-complexity networks with high generalization capability. FMS works by minimizing both training error and required weight precision. According to our theoretical analysis the hidden layer of an FMS-trained autoassociator attempts at coding each input by a sparse code with as few simple features as possible. In experiments the method extracts optimal codes for difficult versions of the ``noisy bars'' benchmark problem by separating the underlying sources, whereas ICA and PCA fail. Real world images are coded with fewer bits per pixel than by ICA or PCA.