Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers

A new survey of large language models as optimizers identifies three distinct paradigms—direct, tool-augmented, and tool-creating—that are reshaping how mathematical optimization problems are solved, according to a preprint posted on arXiv [1]. The paper, titled “Large Language Models as Optimizers: A Survey of Direct vs. Tool-Augmented Approaches and Their Performance Frontiers,” was submitted on 9 April 2026 [1]. It argues that LLMs are increasingly embedded in optimization workflows, often without the end user’s knowledge, because many real-world decisions reduce to searching for better solutions [1]. The survey categorizes current approaches into three paradigms. Direct optimization uses iterative prompting and heuristic generation to navigate solution spaces. Tool-augmented optimization translates natural language problems into formal specifications and orchestrates external solvers. Tool-creating optimization goes further, using LLMs to discover reusable algorithms or heuristics that can be deployed at zero marginal LLM cost [1]. The authors describe performance frontiers drawn from existing benchmarks and identify a critical reasoning gap in current architectures [1]. They argue that direct optimization holds future potential, but tool-augmented optimization offers stronger auditability. Even more powerful future models might favor tool-making to improve operational efficiency for repetitive problem families [1]. The preprint appears on arXiv, an open-access repository founded in 1991 that now receives about 24,000 submissions per month and hosts over two million e-prints [10]. The repository is not peer-reviewed but has become the primary distribution channel in fields such as mathematics, physics, and computer science [10]. The underlying technology that enables LLMs—the transformer architecture—was introduced in 2017 and has since driven the rapid scaling of generative AI applications including chatbots, code generation, and text-to-image models [3][4]. Neural networks, the broader computational framework behind LLMs, consist of layers of artificial neurons that learn hierarchical representations from large datasets, with training accelerated by graphics processing units [6]. The survey’s taxonomy arrives as investment in AI continues to surge. The 2020s AI boom, fueled by transformer-based models, has integrated LLMs into sectors ranging from healthcare and finance to software development and entertainment [3][4]. At the same time, concerns about auditability, energy consumption, and the use of copyrighted training data have prompted debate about the governance of generative AI systems [3]. The paper’s emphasis on the trade-off between direct optimization’s potential and the auditability of tool-augmented methods speaks directly to those concerns [1].