Leo Pape, Faustino Gomez, Mark Ring, Jürgen Schmidhuber. Modular deep belief
networks that do not forget. International Joint Conference
on Neural Networks (IJCNN-2011, San Francisco), 2011.
Abstract
Deep belief networks (DBNs) are popular for learning compact
representations of highdimensional data. However, most approaches so
far rely on having a single, complete training set. If the
distribution of relevant features changes during subsequent training
stages, the features learned in earlier stages are gradually
forgotten. Often it is desirable for learning algorithms to retain
what they have previously learned, even if the input distribution
temporarily changes. This paper introduces the MDBN, an unsupervised
modular DBN that addresses the forgetting problem. M-DBNs are
composed of a number of modules that are trained only on samples
they best reconstruct. While modularization by itself does not
prevent forgetting, the M-DBN additionally uses a learning method
that adjusts each module’s learning rate proportionally to the
fraction of best reconstructed samples. On the MNIST handwritten
digit dataset module specialization largely corresponds to the
digits discerned by humans. Furthermore, in several learning tasks
with changing MNIST digits, MDBNs retain learned features even after
those features are removed from the training data, while monolithic
DBNs of comparable size forget feature mappings learned before.