Does BERT agree? Evaluating knowledge of structure dependence through
agreement relations
release_3542t77ua5c5xkuk3wb3gyg35m
by
Geoff Bacon, Terry Regier
2019
Abstract
Learning representations that accurately model semantics is an important goal
of natural language processing research. Many semantic phenomena depend on
syntactic structure. Recent work examines the extent to which state-of-the-art
models for pre-training representations, such as BERT, capture such
structure-dependent phenomena, but is largely restricted to one phenomenon in
English: number agreement between subjects and verbs. We evaluate BERT's
sensitivity to four types of structure-dependent agreement relations in a new
semi-automatically curated dataset across 26 languages. We show that both the
single-language and multilingual BERT models capture syntax-sensitive agreement
patterns well in general, but we also highlight the specific linguistic
contexts in which their performance degrades.
In text/plain
format
Archived Files and Locations
application/pdf 255.8 kB
file_6webw7pgtbccrbuyv7v7uivcf4
|
arxiv.org (repository) web.archive.org (webarchive) |
1908.09892v1
access all versions, variants, and formats of this works (eg, pre-prints)