BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding release_hzogxb64pzhmbg46hcizf6vyx4

by Abhik Bhattacharjee, Tahmid Hasan, Kazi Samin, M. Sohel Rahman, Anindya Iqbal, Rifat Shahriyar

Entity Metadata (schema)

abstracts[] {'sha1': 'b9ce3ff37e7674f2dcbc1cab90ffba3b0497ce34', 'content': "Pre-training language models on large volume of data with self-supervised\nobjectives has become a standard practice in natural language processing.\nHowever, most such state-of-the-art models are available in only English and\nother resource-rich languages. Even in multilingual models, which are trained\non hundreds of languages, low-resource ones still remain underrepresented.\nBangla, the seventh most widely spoken language in the world, is still low in\nterms of resources. Few downstream task datasets for language understanding in\nBangla are publicly available, and there is a clear shortage of good quality\ndata for pre-training. In this work, we build a Bangla natural language\nunderstanding model pre-trained on 18.6 GB data we crawled from top Bangla\nsites on the internet. We introduce a new downstream task dataset and benchmark\non four tasks on sentence classification, document classification, natural\nlanguage understanding, and sequence tagging. Our model outperforms\nmultilingual baselines and previous state-of-the-art results by 1-6%. In the\nprocess, we identify a major shortcoming of multilingual models that hurt\nperformance for low-resource languages that don't share writing scripts with\nany high resource one, which we name the `Embedding Barrier'. We perform\nextensive experiments to study this barrier. We release all our datasets and\npre-trained models to aid future NLP research on Bangla and other low-resource\nlanguages. Our code and data are available at\n", 'mimetype': 'text/plain', 'lang': 'en'}
contribs[] {'index': 0, 'creator_id': None, 'creator': None, 'raw_name': 'Abhik Bhattacharjee', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 1, 'creator_id': None, 'creator': None, 'raw_name': 'Tahmid Hasan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 2, 'creator_id': None, 'creator': None, 'raw_name': 'Kazi Samin', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 3, 'creator_id': None, 'creator': None, 'raw_name': 'M. Sohel Rahman', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 4, 'creator_id': None, 'creator': None, 'raw_name': 'Anindya Iqbal', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 5, 'creator_id': None, 'creator': None, 'raw_name': 'Rifat Shahriyar', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
ext_ids {'doi': None, 'wikidata_qid': None, 'isbn13': None, 'pmid': None, 'pmcid': None, 'core': None, 'arxiv': '2101.00204v1', 'jstor': None, 'ark': None, 'mag': None, 'doaj': None, 'dblp': None, 'oai': None, 'hdl': None}
files []
filesets []
language en
license_slug ARXIV-1.0
refs []
release_date 2021-01-01
release_stage submitted
release_type article
release_year 2021
title BanglaBERT: Combating Embedding Barrier for Low-Resource Language Understanding
version v1
webcaptures []
work_id qndjkj6fgjdexbnb5oh6cc5lmq

Extra Metadata (raw JSON)

arxiv.base_id 2101.00204
arxiv.categories ['cs.CL']