Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese

K Micallef, A Gatt, M Tanti, L van der Plas, C Borg

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Multilingual language models such as mBERT have seen impressive cross-lingual transfer to a variety of languages, but many languages remain excluded from these models. In this paper, we analyse the effect of pre-training with monolingual data for a low-resource language that is not included in mBERT – Maltese – with a range of pre-training set ups. We conduct evaluations with the newly pre-trained models on three morphosyntactic tasks – dependency parsing, part-of-speech tagging, and named-entity recognition – and one semantic classification task – sentiment analysis. We also present a newly created corpus for Maltese, and determine the effect that the pre-training data size and domain have on the downstream performance. Our results show that using a mixture of pre-training domains is often superior to using Wikipedia text only. We also find that a fraction of this corpus is enough to make significant leaps in performance over Wikipedia-trained models. We pre-train and compare two models on the new corpus: a monolingual BERT model trained from scratch (BERTu), and a further pretrained multilingual BERT (mBERTu). The models achieve state-of-the-art performance on these tasks, despite the new corpus being considerably smaller than typically used corpora for high-resourced languages. On average, BERTu outperforms or performs competitively with mBERTu, and the largest gains are observed for higher-level tasks.
Original languageEnglish
Title of host publicationProceedings of the Third Workshop on Deep Learning for Low Resource NLP (DeepLo'22)
PublisherAssociation for Computational Linguistics
Pages90-101
DOIs
Publication statusPublished - 2022

Fingerprint

Dive into the research topics of 'Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese'. Together they form a unique fingerprint.

Cite this