abstracts[] |
{'sha1': 'b9ce3ff37e7674f2dcbc1cab90ffba3b0497ce34', 'content': "Pre-training language models on large volume of data with self-supervised\nobjectives has become a standard practice in natural language processing.\nHowever, most such state-of-the-art models are available in only English and\nother resource-rich languages. Even in multilingual models, which are trained\non hundreds of languages, low-resource ones still remain underrepresented.\nBangla, the seventh most widely spoken language in the world, is still low in\nterms of resources. Few downstream task datasets for language understanding in\nBangla are publicly available, and there is a clear shortage of good quality\ndata for pre-training. In this work, we build a Bangla natural language\nunderstanding model pre-trained on 18.6 GB data we crawled from top Bangla\nsites on the internet. We introduce a new downstream task dataset and benchmark\non four tasks on sentence classification, document classification, natural\nlanguage understanding, and sequence tagging. Our model outperforms\nmultilingual baselines and previous state-of-the-art results by 1-6%. In the\nprocess, we identify a major shortcoming of multilingual models that hurt\nperformance for low-resource languages that don't share writing scripts with\nany high resource one, which we name the `Embedding Barrier'. We perform\nextensive experiments to study this barrier. We release all our datasets and\npre-trained models to aid future NLP research on Bangla and other low-resource\nlanguages. Our code and data are available at\nhttps://github.com/csebuetnlp/banglabert.", 'mimetype': 'text/plain', 'lang': 'en'}
|
contribs[] |
{'index': 0, 'creator_id': None, 'creator': None, 'raw_name': 'Abhik Bhattacharjee', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 1, 'creator_id': None, 'creator': None, 'raw_name': 'Tahmid Hasan', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 2, 'creator_id': None, 'creator': None, 'raw_name': 'Kazi Samin', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 3, 'creator_id': None, 'creator': None, 'raw_name': 'M. Sohel Rahman', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 4, 'creator_id': None, 'creator': None, 'raw_name': 'Anindya Iqbal', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
{'index': 5, 'creator_id': None, 'creator': None, 'raw_name': 'Rifat Shahriyar', 'given_name': None, 'surname': None, 'role': 'author', 'raw_affiliation': None, 'extra': None}
|
ext_ids |
{'doi': None, 'wikidata_qid': None, 'isbn13': None, 'pmid': None, 'pmcid': None, 'core': None, 'arxiv': '2101.00204v1', 'jstor': None, 'ark': None, 'mag': None, 'doaj': None, 'dblp': None, 'oai': None, 'hdl': None}
|