Policy Improvement via Planning in Maximum Entropy Reinforcement Learning

Abstract

This thesis examines maximum entropy reinforcement learning, an alternative formulation of the traditional reinforcement learning paradigm. Maximum entropy policies prioritize actions leading to states where an agent has more choice between high-value future trajectories. Here we propose two novel algorithms rooted in this framework that use planning-based approaches to improve policy learning. The first model-based algorithm achieves improved value function estimates via Monte Carlo Tree Search. The second algorithm uses a model-free heuristic-based approach to improve deep exploration in challenging environments. Finally, we present a preliminary analysis comparing optimal maximum entropy policies to optimal policies under the traditional reinforcement learning objective.