Induction of Subgoal Automata for Reinforcement Learning
release_h7vthmu54vdr5jkswajeoeckpu
by
Daniel Furelos-Blanco, Mark Law, Alessandra Russo, Krysia Broda and
Anders Jonsson
2019
Abstract
In this work we present ISA, a novel approach for learning and exploiting
subgoals in reinforcement learning (RL). Our method relies on inducing an
automaton whose transitions are subgoals expressed as propositional formulas
over a set of observable events. A state-of-the-art inductive logic programming
system is used to learn the automaton from observation traces perceived by the
RL agent. The reinforcement learning and automaton learning processes are
interleaved: a new refined automaton is learned whenever the RL agent generates
a trace not recognized by the current automaton. We evaluate ISA in several
gridworld problems and show that it performs similarly to a method for which
automata are given in advance. We also show that the learned automata can be
exploited to speed up convergence through reward shaping and transfer learning
across multiple tasks. Finally, we analyze the running time and the number of
traces that ISA needs to learn an automata, and the impact that the number of
observable events has on the learner's performance.
In text/plain
format
Archived Files and Locations
application/pdf 1.5 MB
file_kpxvtodubzfczn4mu7nrzmjgnm
|
arxiv.org (repository) web.archive.org (webarchive) |
1911.13152v1
access all versions, variants, and formats of this works (eg, pre-prints)