Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier

A new framework called PROPEL aims to ease a critical bottleneck in reinforcement learning by training task generators without repeatedly running expensive solver rollouts, according to research published on arXiv [1]. The limiting resource for training agents via reinforcement learning is increasingly the supply of frontier tasks: valid, solvable problems that are just difficult enough to advance the current model [1]. As reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation often yields tasks that are trivial, impossible, or ill-posed [1]. Training a task generator with RL to optimize validity and learnability can address this, but direct optimization requires repeated solver rollouts per candidate. For software-engineering tasks, a single rollout can take tens of minutes, making solver-in-the-loop generator training intractable [1]. PROPEL, introduced by researchers, is a solver-amortized framework that trains a lightweight activation probe on a one-time labeled corpus of generated tasks and solver outcomes [1]. The probe predicts the target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass [1]. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate [1]. For coding, tasks generated at the learnable frontier increased from 10.1% to 20.0% for a Qwen2.5-3B-Instruct solver and from 5.3% to 12.6% for a Qwen2.5-7B-Instruct solver [1]. For software-engineering, PROPEL increased the share of generations at the targeted solve rate from 9.8% to 19.6% for Qwen3.5-27B on repositories not seen during training of the probe and generator [1]. The work builds on the transformer architecture, which was introduced in the 2017 paper "Attention Is All You Need" and has since become the main architecture for large language models [3][4]. Transformers use a multi-head attention mechanism to contextualize tokens within a context window, and they have been widely adopted for training large language models on vast text corpora [4][8]. Unlike earlier recurrent neural architectures such as long short-term memory networks, transformers require less training time because they have no recurrent units [4]. Other generative modeling approaches, such as diffusion models, have also been applied in reinforcement learning contexts, though they are primarily used for computer vision tasks like image generation and denoising [7]. Convolutional neural networks, once the de-facto standard for image processing, have in some cases been replaced by newer architectures such as the transformer [5].