Open but Incompatible: A License Compatibility Analysis of Corpora for Low-Resource African Languages

A new audit of African-language NLP corpora finds that Creative Commons licenses, though widely used, are routinely misapplied, creating legal incompatibilities that can silently prohibit basic tasks such as tokenisation and annotation, according to a paper published on arXiv. The analysis, authored by Ernst van Gassen, examines the license provenance of more than twenty corpus families used in African natural-language processing [1]. It documents four recurring failure modes: outright prohibition, composite license misrepresentation, a NoDerivs clause hidden behind a CC-BY label, and data persistence failure [1]. The JW300 corpus, for example, was removed from the OPUS repository after a legal audit confirmed a violation of its Terms of Service [1]. The WAXAL corpus claims a CC-BY 4.0 license, but that claim is contradicted by its own HuggingFace dataset card [1]. The Tanzil corpus carries a CC-BY label yet its license page explicitly states, “You are not allowed to modify this text in any way,” a NoDerivs restriction that forbids tokenisation and the publication of derived annotation datasets [3]. “The annotation itself is a derivative work,” the paper notes, adding that the prohibition applies regardless of the output license chosen [3]. In the Congolese Radio Corpus, 402 of 405 source URLs are now dead, making the data effectively unreproducible [1]. The paper constructs a six-tier compatibility matrix and applies it to three case-study languages: Kituba/Munukutuba, Zarma, and Moore [1]. Among the concrete findings, the 27Group Feriji corpus is incompatible with the 27Group GEC corpus despite originating from the same research group, and the Leipzig mkw_community_2017 entry—described as the largest open Kituba corpus—has not appeared in any published NLP work [3]. The audit also surfaces a labeling error in the Kongo Wikipedia, where more than 1,200 articles of Kituba text are mislabeled as Kongo, a confusion that extends to the FLORES-200 kon_Latn entry [3]. A separate inventory of African-language datasets on the Hugging Face Hub lists 184 datasets, of which 86 carry an unknown license [6]. The paper recommends that data logging and persistent archiving become standard publication requirements for African NLP work [3].