The information conveyed by the sequence source (the automaton above) can be measured in terms of the entropy of the probability distribution of the possible sequences: . Our next step will be to modify the automaton such that its entropy remains the same but the redundancy among the sequence components increases. All transitions of the form (read: ``go from state to state and emit symbol '') are replaced by a subautomaton consisting of the deterministic sequence of transitions . The modified automaton generates redundant sequences like this one (from class 2):

*aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaeeeeeeeeeeeeeeeeeeee
*

*
ccccccccccccccccccccccccccccccccccccccccdddddddddddddddddddd.
*

This sequence is redundant because many of its components are predictable from other components.

Juergen Schmidhuber 2003-02-19