BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding release_hzogxb64pzhmbg46hcizf6vyx4 [as of editgroup_nwbp5mjbrjdvxgfp6jbpkmjbwq]

by Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin, M. Sohel Rahman, Anindya Iqbal, Rifat Shahriyar

Released as a article .



Pre-training language models on large volume of data with self-supervised objectives has become a standard practice in natural language processing. However, most such state-of-the-art models are available in only English and other resource-rich languages. Even in multilingual models, which are trained on hundreds of languages, low-resource ones still remain underrepresented. Bangla, the seventh most widely spoken language in the world, is still low in terms of resources. Few downstream task datasets for language understanding in Bangla are publicly available, and there is a clear shortage of good quality data for pre-training. In this work, we build a Bangla natural language understanding model pre-trained on 18.6 GB data we crawled from top Bangla sites on the internet. We introduce a new downstream task dataset and benchmark on four tasks on sentence classification, document classification, natural language understanding, and sequence tagging. Our model outperforms multilingual baselines and previous state-of-the-art results by 1-6%. In the process, we identify a major shortcoming of multilingual models that hurt performance for low-resource languages that don't share writing scripts with any high resource one, which we name the `Embedding Barrier'. We perform extensive experiments to study this barrier. We release all our datasets and pre-trained models to aid future NLP research on Bangla and other low-resource languages. Our code and data are available at
In text/plain format

Type  article
Stage   submitted
Date   2021-01-01
Version   v1
Language   en ?
arXiv  2101.00204v1
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Work In Progress

This entity has not been "accepted" into the official database yet.

Catalog Record
State: wip
Revision: a72ad7d6-c957-415d-bb61-aed14aea4d5a