Low-Resource Languages
Low-resource languages are languages for which little digital text, parallel translation data, annotated corpora or other training material exists, so that machine translation and language models perform markedly worse on them than on English and a handful of other well-supported languages. The label describes data availability, not the number of speakers or the linguistic complexity of the language. Of the roughly seven thousand languages spoken today, only a small fraction have enough online text to train a strong model, and the resulting gap in quality is one of the clearest fairness problems in natural language processing.
Data scarcity is the root cause, and it is badly skewed. Joshi and colleagues, in "The State and Fate of Linguistic Diversity and Inclusion in the NLP World" (ACL 2020), grouped languages into six classes by the labeled and unlabeled data available and found that a very large share of the world's languages sit in classes with almost nothing. Web-crawled corpora are dominated by English, and low-resource languages often appear mainly through religious texts, government documents or noisy machine-translated pages. A language can also be low-resource despite many speakers: Swahili, Bengali, Hausa and Yoruba each have tens or hundreds of millions of speakers and still far less usable data than much smaller European languages.
FLORES and the NLLB project were built to measure and attack the problem together. FLORES-101, introduced by Goyal and colleagues at Meta in 2022, provided the same set of professionally translated sentences in 101 languages, so that translation quality could be compared across many directions, including between two low-resource languages without going through English. The FLORES-200 extension covers about 200 languages and is the evaluation set of "No Language Left Behind: Scaling Human-Centered Machine Translation" (NLLB Team, 2022), whose NLLB-200 model translates among 200 languages. The team mined parallel data automatically, used mixture-of-experts layers to share capacity while limiting interference, and reported substantial average gains over previous systems on the benchmark, with the models released openly. Grassroots efforts such as Masakhane, a pan-African research community started in 2019, have contributed datasets and evaluation by speakers of the languages themselves.
Tokenization quietly adds a cost disparity. Tokenizers learned mostly from English text break text in other languages into many more pieces. Petrov, Malfa, Torr and Bibi ("Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023) found that the same text translated into different languages could need up to about fifteen times as many tokens, and Ahia and colleagues reported similar patterns in "Do All Languages Cost the Same?" (EMNLP 2023). Because commercial APIs bill per token and context windows are measured in tokens, speakers of those languages pay more for less usable context, and models see fewer sentences per training step.
Several misconceptions and pitfalls recur. Multilingual training helps transfer, but Conneau and colleagues noted a curse of multilinguality in XLM-R (2020): adding languages to a fixed-capacity model can dilute quality. Translated benchmarks inherit translationese and cultural mismatch, and evaluating a language only through translated English tasks may miss what its speakers care about. Synthetic data, such as back-translation introduced to neural translation by Sennrich and colleagues in 2016, helps but can amplify errors. Some languages are mostly oral or use several scripts, so more text is not always obtainable, and data collection raises consent and ownership questions for communities.
Read low-resource languages as a measure of how unevenly the digital world has recorded human speech: the shortfall is in data, benchmarks and tokenizer efficiency rather than in the languages themselves, and projects such as FLORES, NLLB and community-led datasets are the early, incomplete attempts to correct it.