Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

A new study finds that current multilingual evaluations for Vision-Language Models (VLMs) overlook users of multi-script languages, creating a systematic performance gap between scripts for the same language [1]. The research, posted to the arXiv preprint repository on June 15, introduces PuMVR, a benchmark of 1,000 strictly parallel image-text instances across Punjabi’s three active scripts: Gurmukhi, Shahmukhi, and Roman [1]. The authors evaluated 10 state-of-the-art VLMs and documented a substantial “Script Gap,” where models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16% [1]. Visual input boosted absolute performance uniformly but did not close the orthographic gap [1]. Cross-script in-context transfer was also found to be highly brittle, exposing what the researchers call script-locked knowledge representation [1]. The findings were supported by McNemar tests across all script pairs [1]. Punjabi is spoken by tens of millions of people and is written in Gurmukhi, Shahmukhi, and Roman scripts, making it a representative case for multi-script languages that challenge the one-to-one language-orthography assumption baked into most VLM evaluations [1]. To quantify the problem, the researchers propose the Script Consistency Rate, or SCR, which fell as low as 24.8% on their benchmark, and argue it should become a mandatory metric for script-agnostic evaluation to ensure equitable AI access [1]. The paper was posted on arXiv, an open-access repository of electronic preprints that, as of late 2024, receives about 24,000 new articles per month [6]. arXiv hosts papers in fields including computer science, mathematics, and physics, and is not peer-reviewed [6]. The repository surpassed two million articles by the end of 2021 [6]. The study’s code and data are publicly available on GitHub [1].