M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
release_kkbuw6qqqbe55nxvz6rqn5d2su
by
Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, Nan Duan
2021
Abstract
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that
combines multilingual pre-training and multimodal pre-training into a unified
framework via multitask pre-training. Our goal is to learn universal
representations that can map objects occurred in different modalities or texts
expressed in different languages into a common semantic space. In addition, to
explicitly encourage fine-grained alignment between images and non-English
languages, we also propose Multimodal Code-switched Training (MCT) to combine
monolingual pre-training and multimodal pre-training via a code-switch
strategy. Experiments are performed on the multilingual image retrieval task
across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve
comparable results for English and new state-of-the-art results for non-English
languages.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2006.02635v2
access all versions, variants, and formats of this works (eg, pre-prints)