INTRODUCTION

The following methods for supervised sequence learning have been proposed: Simple recurrent nets [6][2], time-delay nets (e.g. [1]), sequential recursive auto-associative memories [13], back-propagation through time or BPTT [16] [24] [26], Mozer's `focused back-prop' algorithm [9], the IID- or RTRL-algorithm [14][27], its recent improvement [20], the recent fast-weight algorithm [22], higher-order networks [4], as well as continuous time methods equivalent to some of the above [11][12][3]. The following methods for sequence learning by reinforcement learning have been proposed: Extended REINFORCE algorithms [25], the neural bucket brigade algorithm [17], and recurrent networks adjusted by adaptive critics [18](see also [7]).

Common to all of these approaches is that they do not try to selectively focus on relevant inputs; they waste efficiency and resources by focussing on every input. With many applications, a second drawback of these methods is the following: The longer the time lag between an event and the occurrence of a related error the less information is carried by the corresponding error information wandering `back into time' (see [5] for a more detailed analysis). [10] and [15] have addressed the latter problem but not the former.

Back to Recurrent Neural Networks page