Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
release_sehx4775mrhmngcf4ywnr7ydce
by
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, Yu Su
2021
Abstract
Existing studies on question answering on knowledge bases (KBQA) mainly
operate with the standard i.i.d assumption, i.e., training distribution over
questions is the same as the test distribution. However, i.i.d may be neither
reasonably achievable nor desirable on large-scale KBs because 1) true user
distribution is hard to capture and 2) randomly sample training examples from
the enormous space would be highly data-inefficient. Instead, we suggest that
KBQA models should have three levels of built-in generalization: i.i.d,
compositional, and zero-shot. To facilitate the development of KBQA models with
stronger generalization, we construct and release a new large-scale,
high-quality dataset with 64,331 questions, GrailQA, and provide evaluation
settings for all three levels of generalization. In addition, we propose a
novel BERT-based KBQA model. The combination of our dataset and model enables
us to thoroughly examine and demonstrate, for the first time, the key role of
pre-trained contextual embeddings like BERT in the generalization of KBQA.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2011.07743v4
access all versions, variants, and formats of this works (eg, pre-prints)