Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

Researchers have demonstrated a method to manipulate the ranking output of large language models used in information retrieval, exposing a vulnerability in systems that increasingly rely on these models to order search results. [1] The technique, called Rank Anything First (RAF), crafts small, natural-sounding text prompts that can consistently push a chosen item to the top of an LLM-generated list without appearing obviously manipulated. [1][2] The work was submitted to the arXiv preprint repository on October 8, 2025, by Xiyang Hu and colleagues, and a revised version followed on June 28, 2026. [1] arXiv, which began in 1991, hosts over two million scholarly preprints and receives roughly 24,000 new articles each month, serving as a primary distribution channel for computer science research. [6] RAF operates in two stages. The first stage uses a Greedy Coordinate Gradient approach to identify candidate tokens by balancing a ranking-effectiveness gradient against a readability score. [2] The second stage evaluates those candidates under exact ranking and readability losses, employing an entropy-based dynamic weighting scheme, and selects a token through temperature-controlled sampling. [2] The prompt is built token-by-token, guided by the dual objectives of maximizing the target item's rank and preserving linguistic naturalness. [2] Experiments across multiple LLMs showed that RAF significantly boosts the rank of target items using naturalistic language, and the authors report it is more robust than existing methods at both promoting items and maintaining naturalness. [2] The findings highlight that LLM-based reranking is susceptible to adversarial manipulation, raising concerns for the trustworthiness of modern retrieval systems. [1][2] The paper appears on arXiv under the Computation and Language category. The initial submission was 428 KB, and the revised version grew to 438 KB. [1] The authors have released their code on GitHub. [2] The research arrives as the arXiv platform continues to expand its community collaboration tools through arXivLabs, a framework launched in 2020 that allows third-party developers to build experimental features on top of the repository while adhering to principles of openness and user data privacy. [5]