Second term favors few, separated, common component functions.

The term

$$T2 \ := \ W \log \sum_{k \in O} \left( \sum_{i,j \in O \t... ...ial y^k}{\partial w_{ij}}\right)^{2}}} \right)^{2} \mbox{ ,}$$

punishes units with similar influence on the output. We reformulate it:

$$T2 \ = \ W \log \left( \sum_{i,j \in O \times H \cup H \t... ...partial y^k}{\partial w_{uv}}\right)^{2}}} \right) \mbox{ .}$$

Using

$$\frac{\partial y^k}{\partial w_{ij}} = \frac{\partial y^k}{\partial y^i} \frac{\partial y^i}{\partial w_{ij}},$$

this can be rewritten as

$$T2 \ = \ W \log \left( \sum_{i,j \in O \times H \cup H \t... ...c{\partial y^k}{\partial y^u}\right)^{2}}} \right) \mbox{ .}$$

For $i \in O$

$$\frac{\left\vert\frac{\partial y^k}{\partial y^i}\right\vert... ...in O} \left(\frac{\partial y^k}{\partial y^i}\right)^{2}}} =1$$

holds. We obtain

$$T2 \ = \ W \log \left( \left\vert O\right\vert \ left\ve... ...c{\partial y^k}{\partial y^u}\right)^{2}}} \right) \mbox{ .}$$

We observe: (1) an output unit that is very sensitive with respect to two given hidden units will heavily contribute to $T2$ (compare the numerator in the last term in the brackets of $T2$). (2) This large contribution can be reduced by making both hidden units have large impact on other output units (see denominator in the last term in the brackets of $T2$).

Choice of component functions (CFs). FMS tries to figure out a way of using (1) as few CFs as possible for each output unit (this leads to separation of CFs), while simultaneously (2) using the same CFs for as many output units as possible (common CFs).

Back to Independent Component Analysis page.