The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

Pruned large language models can pass multiple-choice evaluations while failing to answer the same question in open generation, according to a study submitted on 16 June 2026. The finding exposes what researchers call a “benchmark illusion” that overstates the usability of compressed models [1]. The study, posted to arXiv, examines how high-sparsity pruning — particularly a method called Wanda — affects multilingual question answering. Compression techniques reduce memory use and inference cost, but the authors show they can also introduce failures that standard benchmarks miss [1]. Under greedy decoding, pruned models often fail to produce the correct answer in open generation, even though they still select it when presented with multiple-choice options [1]. The researchers tracked identical questions before and after pruning. They found that the correct answer is usually not erased but demoted in the model’s output ranking. It often reappears when the decoding strategy is changed to beam search, sampling, or when a single in-context example is provided [1]. The paper describes these as “recognition-only errors,” where the model can recognize the right answer but cannot produce it as the top output under greedy conditions [1]. The work builds on a broader concern in machine learning evaluation: that aggregate benchmark scores can conceal specific failure modes. Prior research in other domains, such as catalyst informatics, has explored how models trained on one dataset may or may not transfer to related tasks, underscoring the importance of testing beyond a single metric [4]. The new findings extend that caution to language model compression, arguing that multiple-choice benchmarks create an evaluation blind spot [1]. The authors recommend that compressed models be tested on what they can produce, not only on what they can recognize. The study does not include external quotes from researchers, but its conclusions are drawn from systematic experiments across languages and pruning levels [1]. The paper was submitted on 16 June 2026 and is available on arXiv [1].