When both and are feedforward networks, the technique proposed above is limited to only certain types of time-varying behavior. With being a sum-and-squash function, the only kind of interesting time-varying output that can be produced is in response to variations in the input; in particular, autonomous dynamical behavior like oscillations (e.g. [Williams and Zipser, 1989]) cannot be performed while the input is held fixed.
It is straight-forward to extend the system above to the case where both and are recurrent. In the experiment below and are non-recurrent, mainly to demonstrate that even a feed-forward system employing the principles above can solve certain tasks that only recurrent nets were supposed to solve.
The method can be accelerated by a procedure analogous to the one presented in [Schmidhuber, 1991b].