AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

A research team has released AfriqueLLM, a suite of open large language models adapted to 20 African languages through continued pre-training on 26 billion tokens, aiming to narrow the performance gap between open and proprietary systems for low-resource languages [1]. The work, led by Hao Yu and colleagues, systematically examines how data composition and model architecture affect language adaptation. The study spans five base models across three families — Llama 3.1, Gemma 3, and Qwen 3 — and finds that data composition is the primary driver of continued pre-training gains [1]. Adding math, code, and synthetic translated data to the training mixture yielded consistent improvements, including on reasoning-oriented evaluations [1]. Open large language models continue to underperform relative to proprietary systems, with the gap most pronounced for African languages [2]. Continued pre-training offers a practical route to language adaptation, but improvements on demanding capabilities such as mathematical reasoning often remain limited due to uneven domain coverage and missing task-relevant knowledge in low-resource language corpora [2]. The researchers found that within a fixed architecture, larger models typically improve performance, but architectural choices dominate scale when comparing across model families [1]. Strong multilingual performance in a base model did not reliably predict post-continued-pre-training outcomes; robust architectures paired with task-aligned data provided a more dependable recipe [1]. The best models also improved long-context performance, including document-level translation [1]. Qwen, one of the base model families used in the study, is developed by Alibaba Cloud and distributed under open-source licenses including Apache 2.0 [9]. Large language models are machine learning models with many parameters, trained with self-supervised learning on vast amounts of text [8]. The AfriqueLLM models and code have been released on Hugging Face and GitHub [1]. Hugging Face's platform allows researchers to link papers to models, datasets, and interactive demos, and supports arXiv integration that embeds demos directly alongside paper abstracts [4][5]. The release comes amid broader industry attention to cost-efficient open-weight models. Chinese firm DeepSeek, for instance, reported training its V3 model for approximately $6 million, a fraction of the cost of comparable proprietary systems, using open-weight releases under the MIT License [7].