We compare LOCOCODE to ``independent component analysis'' (ICA, e.g., [5,1,4,19]) and ``principal component analysis'' (PCA, e.g., [21]). ICA is realized by Cardoso's JADE algorithm, which is based on whitening and subsequent joint diagonalization of 4th-order cumulant matrices. To measure the information conveyed by resulting codes we train a standard backprop net on the training set used for code generation. Its inputs are the code components; its task is to reconstruct the original input. The test set consists of 500 off-training set exemplars (in the case of real world images we use a separate test image). Coding efficiency is the average number of bits needed to code a test set input pixel. The code components are scaled to the interval and partitioned into discrete intervals. Assuming independence of the code components we estimate the probability of each discrete code value by Monte Carlo sampling on the training set. To obtain the test set codes' bits per pixel (Shannon's optimal value) the average sum of all negative logarithms of code component probabilities is divided by the number of input components. All details necessary for reimplementation are given in [15].
Noisy bars adapted from [11,12]. The input is a pixel grid with horizontal and vertical bars at random positions. The task is to extract the independent features (the bars). Each of the 10 possible bars appears with probability . In contrast to [11,12] we allow for bar type mixing -- this makes the task harder. Bar intensities vary in ; input units that see a pixel of a bar are activated correspondingly others adopt activation . We add Gaussian noise with variance 0.05 and mean 0 to each pixel. For ICA and PCA we have to provide information about the number (ten) of independent sources (tests of versions with assumed sources will be denoted by ICA- and PCA-). LOCOCODE does not require this -- using 25 hidden units (HUs) we expect LOCOCODE to prune the 15 superfluous HUs.
Results. See Table 1. While the reconstruction errors of all methods are similar, LOCOCODE has the best coding efficiency. 15 of the 25 HUs are indeed automatically pruned: LOCOCODE finds an optimal factorial code which exactly mirrors the pattern generation process. PCA codes and ICA-15 codes, however, are unstructured and dense. While ICA-10 codes are almost sparse and do recognize some sources, the sources are not clearly separated like with LOCOCODE -- compare the weight patterns shown in [15].
Real world images. Now we use more realistic input data, namely subsections of: 1) the aerial shot of a village, 2) an image of wood cells, and 3) an image of striped piece of wood. Each image has pixels, each taking on one of 256 gray levels. ( for village) pixels subsections are randomly chosen as training inputs. Test sets stem from images similar to 1), 2), and 3).
Results. For the village image LOCOCODE discovers on-center-off-surround hidden units forming a sparse code. For the other two images LOCOCODE also finds appropriate feature detectors -- see weight patterns shown in [15]. Using its compact, low-complexity features it always codes more efficiently than ICA and PCA.