Attention-Guided Answer Distillation for Machine Reading Comprehension
release_elvgrytac5ci7lwfaosua6wiey
by
Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang,
Ming Zhou
2018
Abstract
Despite that current reading comprehension systems have achieved significant
advancements, their promising performances are often obtained at the cost of
making an ensemble of numerous models. Besides, existing approaches are also
vulnerable to adversarial attacks. This paper tackles these problems by
leveraging knowledge distillation, which aims to transfer knowledge from an
ensemble model to a single model. We first demonstrate that vanilla knowledge
distillation applied to answer span prediction is effective for reading
comprehension systems. We then propose two novel approaches that not only
penalize the prediction on confusing answers but also guide the training with
alignment information distilled from the ensemble. Experiments show that our
best student model has only a slight drop of 0.4% F1 on the SQuAD test set
compared to the ensemble teacher, while running 12x faster during inference. It
even outperforms the teacher on adversarial SQuAD datasets and NarrativeQA
benchmark.
In text/plain
format
Archived Files and Locations
application/pdf 725.1 kB
file_zva3dtma4fdpzlwvp3izwkj73m
|
arxiv.org (repository) web.archive.org (webarchive) |
1808.07644v2
access all versions, variants, and formats of this works (eg, pre-prints)