"Chi nas dal soch el sent de legn" -- Auditing Text Corpora for Lombard

A manual audit of text corpora for Lombard, an under-resourced language continuum from Italy, has found that web-scraped datasets are riddled with language misidentification, boilerplate text, and non-linguistic noise, according to a new study [1]. The study, posted to arXiv on June 4, 2026, examined parallel and monolingual corpora available for Lombard [1]. Researchers found that the perceived abundance of web-scraped data is an illusion, with massive datasets plagued by severe quality issues [1]. The analysis further revealed that high-quality data is heavily skewed towards Western Lombard varieties, while Eastern Lombard varieties are left on the margins [1]. The authors underscore the need for variety-aware, community-driven data curation rather than purely quantity-driven scraping [1]. The findings highlight a broader challenge in natural language processing for low-resource languages. The lack of high-quality datasets impedes the training, development, and evaluation of systems for tasks such as machine translation [1]. The study also analyzed the orthographic composition of valid Lombard portions across web-scraped datasets, curated corpora, and benchmarks, finding conflicting orthographical systems and severe representational bias across all corpora [1]. While the Lombard audit focuses on data quality, related research on large language models has documented systematic reasoning failures when models encounter counterintuitive problems. A controlled benchmarking study on discrete probability problems found that eight state-of-the-art models achieved an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones [4]. The same study provided empirical evidence of token bias, with performance dropping by over 20% when canonical formulations were replaced by disguised variants [4]. Embedding misleading suggestions in the prompt reduced performance by up to 34%, with no model proving immune [4]. These findings suggest that current LLMs are not yet genuine probabilistic reasoners, despite their success in advanced mathematical problems [4]. The Lombard corpus audit adds to a growing body of work examining the limitations of automated data collection and model evaluation. The researchers call for community involvement to ensure that underrepresented language varieties receive adequate attention in dataset construction [1].