Is Code Better Than Language for Algorithmic Reasoning

Deterministic code execution outperforms natural-language reasoning by 31.6 percentage points on algorithmic tasks, but changing only the intermediate representation — without actual execution — yields no meaningful gain, according to a new study on arXiv. The paper, submitted June 14, disentangles two factors often conflated when comparing tool-augmented language models: the intermediate representation (code vs. natural language) and the execution mechanism [1]. On a 40-task verifiable algorithmic benchmark, deterministic code execution delivered a +31.6pp improvement over natural-language reasoning [1]. However, an intermediate intervention — where the model expressed reasoning as executable code but the language model simulated that code in context — was not meaningfully different from natural-language reasoning, showing a difference of just +0.15pp [1]. These results suggest that changing the intermediate representation alone does not explain the tool-use advantage, providing evidence that performance gains require reliable external execution [1]. The authors formalize this intuition with a statistical decision-theoretic model that characterizes when execution dominates end-to-end risk in the disentangled trace-generation/execution regime [1]. A reconstruction intervention, which used a proxy language model to infer natural-language reasoning traces from code representations, recovered performance comparable to the original natural-language reasoning pipeline, further validating the theory [1]. Large language models, the neural networks underlying modern chatbots, are typically based on transformer architectures and are pre-trained to predict the next word before being fine-tuned to follow instructions [2]. A reasoning model is a type of LLM specifically trained to solve complex tasks requiring multiple steps of logical reasoning, demonstrating superior performance on logic, mathematics, and programming tasks compared to standard LLMs [5]. The new study's findings arrive as the practice of "vibe coding" — where developers describe a task to an LLM and accept AI-generated code without thorough review — gains traction. The term was coined in February 2025 by computer scientist Andrej Karpathy and was named the Collins English Dictionary Word of the Year for 2025 [3]. Critics of the practice point to a lack of accountability, maintainability, and an increased risk of security vulnerabilities [3]. All experiments associated with the paper are hosted on Hugging Face, a platform that has collaborated with arXiv since 2022 to embed interactive machine-learning demos directly alongside papers [7][8]. The integration allows users to find open-source demos on a paper's arXiv abstract page and try them in a browser without writing code [9].