Previous autoassociators (AAs). Backprop-trained AAs without a narrow hidden bottleneck (``bottleneck'' refers to a hidden layer containing fewer units than other layers) typically produce redundant, continuous-valued codes and unstructured weight patterns. Baldi and Hornik (1989) studied linear AAs with a hidden layer bottleneck and found that their codes are orthogonal projections onto the subspace spanned by the first principal eigenvectors of a covariance matrix associated with the training patterns. They showed that the mean squared error (MSE) surface has an unique minimum. Nonlinear codes have been obtained by nonlinear bottleneck AAs with more than 3 (e.g., 5) layers, e.g., Kramer (1991), Oja (1991) or DeMers and Cottrell (1993). None of these methods produces sparse, factorial or local codes -- instead they produce first principal components or their nonlinear equivalents (``principal manifolds''). We will see that FMS-based AAs yield quite different results.
FMS-based AAs. According to subsections 3.1 and 3.2, because of the low-complexity coding aspect the codes tend to (C1) be binary for sigmoid units with activation function ( is small for near 0 or 1), (C2) require few separated code components or hidden units (HUs), and (C3) use simple component functions. Because of the low-complexity decoding part, codes also tend to (D1) have many HUs near zero and, therefore, be sparsely (or even locally) distributed, (D2) have code components conveying information useful for generating as many output activations as possible. (C1), (C2) and (D2) encourage minimally redundant, binary codes. (C3), (D1) and (D2), however, encourage sparse distributed (local) codes. (C1) - (C3) and (D1) - (D2) lead to codes with simply computable code components (C1, C3) that convey a lot of information (D2), and with as few active code components as possible (C2, D1). Collectively this makes code components represent simple input features.