JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory

Researchers have proposed two new frameworks, JE-IRT and Multilingual-IRT, to improve the evaluation of Large Language Models (LLMs). JE-IRT embeds LLMs and questions in a shared geometric space, while Multilingual-IRT assesses LLMs across languages.

The JE-IRT framework, presented in a paper submitted to arXiv on September 26, 2025^[1], and later revised on June 14, 2026, embeds both LLMs and questions in a shared geometric space. This allows for a more nuanced evaluation of LLMs, replacing global rankings with topical specialization. The framework enables smooth variation across related questions and reveals an LLM-internal taxonomy that partially aligns with human-defined subject categories. The paper notes that larger norms in the embedding space consistently indicate harder questions^[1]. A separate study, also on arXiv^[2], introduced Multilingual-IRT, a statistical framework designed to efficiently evaluate LLMs across languages. Multilingual-IRT addresses issues with current multilingual benchmarks, such as exhaustive evaluation scales and automatic translation errors, by extending Item Response Theory with per-language difficulty deviations and split discriminability. The framework was fitted on 25 LLMs across 29 languages of MMLU-Pro-X, with 28 being non-English languages^[2]. The study found that the percentage of lower binary cross-entropy was between 11-16%^[2].