Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics

Researchers have found that the position of a query in In-Context Learning for Diffusion Large Language Models significantly impacts generation quality, and introduced a new decoding framework, S2D2, to improve accuracy and speed.

A recent study published on arXiv^[1] revealed that Diffusion Large Language Models (dLLMs), unlike Autoregressive models, utilize bidirectional attention, offering flexibility in query placement. However, current practices still follow Autoregressive-style trailing-query templates. The study demonstrated that query position is a crucial variable in dLLMs, affecting generation quality similarly to example semantic quality. To address this, the researchers introduced Auto-ICL, a training-free adaptive routing strategy that optimizes query placement. Another study on arXiv^[2] introduced S2D2, a hybrid decoding trajectory combining diffusion and autoregressive modes, which achieves up to 4.7x speedup over autoregressive decoding on SDAR^[2]. S2D2 can be used with three mainstream block-diffusion families and remains complementary to built-in self-correction on LLaDA2.1-Mini. The findings suggest that S2D2 improves the accuracy-speed tradeoff over strong confidence-thresholding baselines.