next up previous
Next: How Many Sub-Goals for Up: Learning to Generate Sub-Goals Previous: The basic approach

Cascaded sub-goaling with a recurrent sub-goal generator

Instead of using different sub-goal generators for different numbers of sub-goals, we can change our basic architecture such that only one sub-goal generator is necessary for generating arbitrary numbers of sub-goals.

The idea is to take a recurrent sub-goal generator which at a given time step produces only one sub-goal. At the next time step this sub-goal is fed back to the start input of the same sub-goal generator (while the goal input remains constant). To adjust the weights of the sub-goal generator, we can use an algorithm inspired by the `back-propagation through time'-method: Successive sub-goals have to be fed into copies of $E$ as shown in figure 3 (figure 3 shows the special case of three sub-goals). Gradient descent requires to change $W_S$ according to the sum of all gradients computed for the various copies of $S$. ( Of course, $E$'s weight vector has to remain constant during $S$' credit assignment phase.)

While unfolding the $S/E$ system in time, it is not necessary to build real copies of $E$ and $S$. It suffices if during activation spreading each unit in $E$ and $C$ stores its time-varying activations on a stack, from which they are popped during back-propagation phase.



Subsections
next up previous
Next: How Many Sub-Goals for Up: Learning to Generate Sub-Goals Previous: The basic approach
Juergen Schmidhuber 2003-03-14

Back to Subgoal learning - Hierarchical Learning
German pages with Subgoal learning pictures