Controlling Conditional Language Models without Catastrophic Forgetting
release_g4vyl3dcrjerzgqg637ohora2y
by
Tomasz Korbak and Hady Elsahar and German Kruszewski and Marc Dymetman
2022
Abstract
Machine learning is shifting towards general-purpose pretrained generative
models, trained in a self-supervised manner on large amounts of data, which can
then be applied to solve a large number of tasks. However, due to their generic
training methodology, these models often fail to meet some of the downstream
requirements (e.g., hallucinations in abstractive summarization or style
violations in code generation). This raises the important question of how to
adapt pre-trained generative models to meet all requirements without destroying
their general capabilities ("catastrophic forgetting"). Recent work has
proposed to solve this problem by representing task-specific requirements
through energy-based models (EBMs) and approximating these EBMs using
distributional policy gradients (DPG). Despite its effectiveness, this approach
is however limited to unconditional distributions. In this paper, we extend DPG
to conditional tasks by proposing Conditional DPG (CDPG). We evaluate CDPG on
four different control objectives across three tasks (translation,
summarization and code generation) and two pretrained models (T5 and GPT-Neo).
Our results show that fine-tuning using CDPG robustly moves these pretrained
models closer towards meeting control objectives and -- in contrast with
baseline approaches -- does not result in catastrophic forgetting.
In text/plain
format
Archived Files and Locations
application/pdf 2.6 MB
file_p5lveci24fdc7jmucpdhoi5exq
|
arxiv.org (repository) web.archive.org (webarchive) |
2112.00791v2
access all versions, variants, and formats of this works (eg, pre-prints)