EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning

Researchers have proposed EnvRL, a reinforcement learning framework that improves how large language model agents handle long-horizon tasks by learning environment dynamics from their own interactions, according to a paper posted on arXiv [1]. The framework addresses a known limitation in conventional reinforcement learning (RL) for agentic tasks: sparse outcome rewards often fail to capture the rich information embedded in rollout trajectories [1]. EnvRL incorporates two auxiliary objectives—state prediction and inverse dynamics—alongside the primary RL objective. The authors argue that interaction experience “serves as an implicit supervision signal, reveals the underlying transition mechanisms of the environment, and enables the agent to construct a more accurate internal model of the environment” [1]. In experiments, the team applied EnvRL to the Qwen-2.5-1.5B-Instruct model using the GRPO algorithm [1]. On the ALFWorld benchmark, the success rate rose from 72.8% to 77.4%. On WebShop, the rate climbed from 56.8% to 67.0% [1]. Both benchmarks involve long-horizon, multi-step tasks where agents must navigate environments and make sequential decisions. Large language models, or LLMs, are machine learning models with many parameters trained on vast text corpora using self-supervised learning [8]. Their use as autonomous agents has expanded rapidly, but training them with RL for extended tasks remains difficult because feedback is often delayed or sparse [1]. EnvRL’s approach of jointly optimizing policy and environment dynamics aims to give the agent a richer training signal without requiring additional external data. The paper was submitted to arXiv on June 16, 2026 [1]. arXiv, founded in 1991, is an open-access repository for electronic preprints in fields such as computer science, mathematics, and physics [6]. As of late 2024, it receives roughly 24,000 new submissions per month and has surpassed two million total articles [6]. Papers on the platform are moderated but not peer-reviewed [6]. The repository also supports community-built tools through arXivLabs, a framework that allows third-party developers to create features such as citation explorers and code finders that appear on article pages [4][5].