With a given set of patterns,
the goal of redundancy reduction without loss of information is this:
Generate a new set of pattern vectors (the code) such that (a) for
each pattern from the original set
there is exactly one pattern
from the code
( *represents* ) and (b) the
components of code patterns are ``less dependent on each other''.
To make (b) more precise, let us measure the
degree of the redundancy of a random variable
with components by

where

denotes the entropy of variable . Ideally, the result of redundancy reduction is a ``factorial code''. With a factorial code, the code components are statistically independent:

(throughout this paper, the subscript of a symbol representing a vector indexes the -th component of the vector).

Juergen Schmidhuber 2003-02-19