When both and are feedforward networks, the technique proposed above is limited to only certain types of time-varying behavior. With being a sum-and-squash function, the only kind of interesting time-varying output that can be produced is in response to variations in the input; in particular, autonomous dynamical behavior like oscillations (e.g. [Williams and Zipser, 1989]) cannot be performed while the input is held fixed.

It is straight-forward to extend the system above to the
case where both and are recurrent.
In the experiment below and are *non-recurrent*,
mainly to demonstrate that even a feed-forward system employing
the principles above can solve certain tasks that only recurrent nets
were supposed to solve.

The method can be accelerated by a procedure analogous to the one presented in [Schmidhuber, 1991b].

Juergen Schmidhuber 2003-02-13

Back to Recurrent Neural Networks page