Partially Observable Hierarchical Reinforcement Learning with AI Planning

Abstract

Partially observable Markov decision processes (POMDPs) challenge reinforcement learning agents due to incomplete environment states. Even assuming monotonicity in uncertainty, it is difficult for an agent to know how and when to stop exploring the environment for a given task. In this abstract, we discuss how to use hierarchical reinforcement learning (HRL) and AI Planning (AIP) to improve exploration when the agent knows possible valuations of unknown predicates and how to discover them. By encoding the uncertainty in an abstract planning model, the agent can derive a high-level plan which is then used to decompose the overall POMDP into a tree of semi-POMDPs for training. We evaluate our agent’s performance on the MiniGrid domain and show how guided exploration may improve agent performance.