G-IdiomAlign: A Gloss-Pivoted Benchmark for Cross-Lingual Idiom Alignment

A new benchmark called G-IdiomAlign aims to measure how well large language models handle one of translation's thorniest problems: idioms. The system uses English glosses from Wiktionary to anchor non-literal phrases across languages, revealing that models still default to literal translations far too often. The benchmark, described in a paper submitted in 2026, addresses the long-standing difficulty of transferring idioms between languages. Idioms resist direct translation because their meaning is not built from the individual words they contain, a property researchers call non-compositionality [1][2]. G-IdiomAlign tackles this by pivoting each idiom on an English definition, or gloss, drawn from Wiktionary, and constructing a high-confidence reference alignment set for reproducible testing [1][2]. The benchmark supports two evaluation protocols. The first is a multiple-choice task with typed distractors designed to trace specific error patterns. The second, called Gloss-Contrastive Generation, compares model outputs with and without a gloss to isolate the effect of providing an explicit semantic anchor [1][2]. Across a range of large language models, or LLMs—a class of machine learning models trained on vast text corpora for language generation [8]—the dominant failure mode was a bias toward literal translation, a problem that intensified when the target language had fewer digital resources [1][2]. Glosses consistently improved performance on the generation task when measured by an embedding-based semantic proxy, but overall results remained modest. The authors note this indicates "substantial headroom in the open output space" [2]. A deeper analysis focused on Qwen3-8B, a model from the Qwen family developed by Alibaba Cloud [9]. The researchers found that differences between the gloss and no-gloss conditions were concentrated more in the model's attention heads than in its layers, and that better generations with glosses coincided with stronger anchoring on the provided definition [1][2]. The work arrives as the machine learning community increasingly links papers to executable demos and code. Platforms like Hugging Face now allow researchers to create paper pages that aggregate models, datasets, and interactive Spaces, and even embed demos directly on arXiv abstract pages [4][5]. While no interactive demo for G-IdiomAlign was immediately available, the paper's presence on arXiv enables such community-built interfaces to be linked in the future [5].