Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces

Researchers have introduced OPT*, a family of tasks to improve step-by-step optimization-like reasoning in large language models (LLMs), and developed a method using knowledge graphs and LLMs to predict gene knockout perturbations on gene expression.

OPT* is designed to train and evaluate LLMs on optimization-style tasks, providing a feasibility checker and evaluator, with a complexity parameter that expands the search space without requiring new human labels^[1]. The tasks are studied in two regimes: solver-guided online policy optimization and search-based offline RL. Verifiable reward training has already shown improvements in mathematical and coding reasoning. Meanwhile, a new method leveraging biological knowledge graphs and reasoning LLMs has been developed to predict the effect of gene knockout perturbations on transcriptomic gene expression. This method has achieved highly competitive performance on out-of-distribution perturbation prediction, outperforming almost all other methods^[2]. The use of knowledge graphs improves extrapolation beyond training perturbations, and reasoning LLMs trained via reinforcement learning further enhance performance.