State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

A new study finds that how market data is presented to a deep reinforcement learning agent is a decisive factor in energy trading performance, with combined feature sets significantly outperforming any single data type [1]. The research, posted to the preprint server arXiv on June 25, 2026, by Vincent Francois-Lavet, used a custom environment called HydroDam to simulate pumped-storage arbitrage [1]. The agent, a fixed Double DQN, was trained on Belgian day-ahead electricity prices from 2007 to 2011 and tested on data from 2012 to 2025, as well as across 39 other European market zones [1]. Reinforcement learning, a machine learning paradigm where an agent learns to maximize a reward signal through interaction with a dynamic environment, has been applied to complex games and scientific problems by labs such as Google DeepMind [3][4]. The study compared three families of market features: absolute price and calendar data, relative features comparing current prices to recent history, and short-horizon forecasts [1]. When the agent was provided only absolute price and calendar features, it achieved just 28.8% of a benchmark profit score on the primary test set and a median of 5.7% across the 39 cross-zone tests [1]. Single-feature-family states using only relative or only forecast data also failed to beat a simple rolling price-score heuristic in the cross-zone median evaluation [1]. Performance improved markedly when feature families were combined. An agent using absolute and relative features reached 49.9% on the test set and a 39.8% cross-zone median [1]. The strongest result came from combining all three families—absolute, relative, and forecast—which yielded a test-set score of 55.6% and a cross-zone median of 47.5% [1]. The paper concludes that state representation is not a minor preprocessing choice but a central part of policy design for storage-trading reinforcement learning, and that robust transfer across markets requires integrating price scale, recent relative context, and forecast information [1].