WHAT IS REDUNDANCY REDUCTION?

With a given set of patterns, the goal of redundancy reduction without loss of information is this: Generate a new set of pattern vectors (the code) such that (a) for each pattern $x^p$ from the original set there is exactly one pattern $y^p$ from the code ($y^p$ represents $x^p$) and (b) the components of code patterns are ``less dependent on each other''. To make (b) more precise, let us measure the degree of the redundancy of a random variable $X$ with components $X_i$ by

$$R(X) = \frac{ \sum_i H(X_i) - H(X)} {H(X)},$$

where

$$H(Z) = - \sum_{all~cases~\alpha} P(Z = \alpha)log P(Z = \alpha)$$

denotes the entropy of variable $Z$. Ideally, the result of redundancy reduction is a ``factorial code''. With a factorial code, the code components are statistically independent:

$$\forall p: ~~ P(x^p) = P(y^p) = \prod_i P(y_i = y^p_i)$$

(throughout this paper, the subscript $i$ of a symbol representing a vector indexes the $i$-th component of the vector).