Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

An audit of 18 multimodal large language models found none were order-invariant, with flip rates spanning 24-50% when the sequence of inputs was shuffled, according to a paper posted to arXiv [1]. The study introduces Facet-Probe, a five-facet audit covering option, evidence-chunk, document-rank, image-set, and mixed-modality ordering [1]. Researchers applied a Bayesian item-response model to separate ordering noise from per-facet bias and used a same-ordering control to estimate the decoder-stochastic floor for observed flips [1]. A Gemini same-ordering control at temperature 0 indicated a substantial ordering excess over a same-input decoder-noise floor in verified cells [1]. Capability predicted but did not eliminate flips; the best model still flipped on 13.4% of trials [1]. Machine learning, the broader field encompassing these models, relies on statistical algorithms that learn from data and generalize to unseen tasks [3]. The paper argues that standard benchmarks score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines [1]. In mitigation tests on Gemini, training-free prompt changes were modality-conditional and did not transfer from text to visual reasoning [1]. The authors conclude that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches [1]. They propose cross-ordering flip rate as a standard reporting axis for MLLMs [1].