Recurrent Policy Gradients Daan Wierstra, Alexander Förster, Jan Peters, Jürgen Schmidhuber Abstract Reinforcement learning for partially observable Markov decision problems (POMDPs) is a challenge as it requires policies with an internal state. Traditional approaches suffer significantly from this shortcoming and usually make strong assumptions on the problem domain such as perfect system models, state-estimators and a Marko- vian hidden system. Recurrent neural networks (RNNs) offer a natural framework for dealing with policy learning using hidden state and require only few limiting assumptions. As they can be trained well using gradient descent, they are suited for policy gradient approaches. In this paper, we present a policy gradient method, the Recurrent Policy Gradient which constitutes a model-free reinforcement learning method. It is aimed at train- ing limited-memory stochastic policies on problems which require long-term mem- ories of past observations. The approach involves approximating a policy gradient for a recurrent neural network by backpropagating return-weighted characteristic eligibilities through time. Using a “Long Short-Term Memory” RNN architecture, we are able to outperform previous RL methods on three important benchmark tasks. Furthermore, we show that using history-dependent baselines helps reduc- ing estimation variance significantly, thus enabling our approach to tackle more challenging, highly stochastic environments. Key words: Recurrent Neural Networks, Policy Gradient Methods, Reinforcement Learning, Partially Observable Markov Decision Problems (POMDPs)