World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models

A new evaluation framework called wmw audits whether vision-language models construct physically coherent explanations when answering questions about scenes, rather than merely scoring final-answer accuracy. The framework, introduced in a paper submitted on 28 May 2026, requires models to produce a typed trace that includes an initial state, a state transition, a resulting state, and an answer [1][2]. A hybrid verifier then checks schema validity, state grounding, transition consistency, and answer-trace compatibility, assigning error labels such as object, relation, force, transition, temporal, unit/scale, and faithfulness errors [1][2]. The authors release Tracebank, a controlled resource containing schema-validated synthetic scenarios across multiple physics families and minimally perturbed contrastive preference pairs, alongside verifier code and audit guidelines [2]. When applied to mid-tier models, wmw exposed that 35% of correct answers were backed by physically invalid traces [1][2]. Verifier-guided reranking recovered up to 7 percentage points of trace validity without reducing answer accuracy, and trace-level preference tuning reduced hidden inconsistency by 41% relative [1][2]. The work does not introduce another final-answer physics benchmark but instead offers a reusable protocol for measuring whether a model's stated physical world can be true at the same time as its answer [2]. Vision-language models are part of a broader artificial intelligence landscape that since the 2020s has seen generative AI become widely available for producing images, audio, and video from text prompts [3]. The wmw framework addresses a gap in how such systems are evaluated: answer-only scoring can obscure whether a model perceived the correct objects, represented the correct physical state, or predicted a plausible transition [1][2]. This concern parallels wider discussions about algorithmic bias, where systematic and repeatable errors can arise from design decisions, training data, or unanticipated use contexts [4]. A 2021 survey identified multiple forms of algorithmic bias, including historical, representation, and measurement biases, each of which can contribute to unfair outcomes [4]. The paper's emphasis on auditability and typed error labels echoes broader movements toward structured evaluation and transparency. In sustainability science, for instance, metrics are evolving and include indicators, benchmarks, audits, and certification systems such as Fairtrade and Organic [5]. The wmw verifier functions as a similar auditing instrument, checking not just whether an answer is correct but whether the reasoning chain that produced it is internally consistent and physically grounded [1][2]. The submission, authored by Emmanuelle Bourigault and hosted on arXiv through the arXivLabs framework, is 155 KB in size [1].