Does BERT agree? Evaluating knowledge of structure dependence through agreement relations release_3542t77ua5c5xkuk3wb3gyg35m

by Geoff Bacon, Terry Regier

Released as a article .

2019  

Abstract

Learning representations that accurately model semantics is an important goal of natural language processing research. Many semantic phenomena depend on syntactic structure. Recent work examines the extent to which state-of-the-art models for pre-training representations, such as BERT, capture such structure-dependent phenomena, but is largely restricted to one phenomenon in English: number agreement between subjects and verbs. We evaluate BERT's sensitivity to four types of structure-dependent agreement relations in a new semi-automatically curated dataset across 26 languages. We show that both the single-language and multilingual BERT models capture syntax-sensitive agreement patterns well in general, but we also highlight the specific linguistic contexts in which their performance degrades.
In text/plain format

Archived Files and Locations

application/pdf  255.8 kB
file_6webw7pgtbccrbuyv7v7uivcf4
arxiv.org (repository)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article
Stage   submitted
Date   2019-08-26
Version   v1
Language   en ?
arXiv  1908.09892v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 14051535-d6b1-4919-b3bb-5b8a34273cc9
API URL: JSON