AN ERROR FUNCTION FOR THE INDEPENDENCE CRITERION

For the sake of argument, let us assume that at all times each $P_i$ is as good as it can be, meaning that $P_i$ always predicts the expectation of $y_i$ conditioned on the outputs of the other modules, $E(y_i \mid \{y^p_k, k \neq i \})$. (In practice, the predictors will have to be retrained continually.) In the case of quasi-binary codes the following objective function $H$ is zero if the independence criterion is met:

$$H = \frac{1}{2} \sum_i \sum_p \left[ P^p_i - \bar{y_i} \right]^2.$$ (2)

This term for mutual predictability minimization aims at making the outputs independent - similar to the goal of a term for maximizing the determinant of the covariance matrix under Gaussian assumptions (Linsker, 1988). The latter method, however, tends to remove only linear predictability, while the former can remove non-linear predictability as well (even without Gaussian assumptions), due to possible non-linearities learnable by non-linear predictors.

Back to Independent Component Analysis page.