Batch Reinforcement Learning from Crowds
release_wkvzmd7xhbb35l2ahzgl7v3nqa
by
Guoxi Zhang, Hisashi Kashima
2021
Abstract
A shortcoming of batch reinforcement learning is its requirement for rewards
in data, thus not applicable to tasks without reward functions. Existing
settings for lack of reward, such as behavioral cloning, rely on optimal
demonstrations collected from humans. Unfortunately, extensive expertise is
required for ensuring optimality, which hinder the acquisition of large-scale
data for complex tasks. This paper addresses the lack of reward in a batch
reinforcement learning setting by learning a reward function from preferences.
Generating preferences only requires a basic understanding of a task. Being a
mental process, generating preferences is faster than performing
demonstrations. So preferences can be collected at scale from non-expert humans
using crowdsourcing. This paper tackles a critical challenge that emerged when
collecting data from non-expert humans: the noise in preferences. A novel
probabilistic model is proposed for modelling the reliability of labels, which
utilizes labels collaboratively. Moreover, the proposed model smooths the
estimation with a learned reward function. Evaluation on Atari datasets
demonstrates the effectiveness of the proposed model, followed by an ablation
study to analyze the relative importance of the proposed ideas.
In text/plain
format
Archived Files and Locations
application/pdf 582.9 kB
file_s4jhui74ofhfblhrntev7h2lti
|
arxiv.org (repository) web.archive.org (webarchive) |
2111.04279v1
access all versions, variants, and formats of this works (eg, pre-prints)