Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning

A new method enables reinforcement learning agents to learn from streaming data in partially observable environments, a setting that had previously confounded online approaches. The technique relies on a diagonal recurrent architecture called recurrent trace units to perform exact real-time recurrent learning with linear complexity [1]. Streaming reinforcement learning requires an agent to process data one sample at a time, without a replay buffer or batched updates. While this paradigm has scaled to deep function approximation under full observability, partially observable settings remained out of reach because truncated backpropagation through time collapses to a one-step gradient horizon in the streaming regime, and exact real-time recurrent learning (RTRL) has been prohibitively expensive [1]. The new work, posted to arXiv on 23 May 2026, closes that gap [1]. The authors employ recurrent trace units (RTUs), a diagonal recurrent architecture that enables exact RTRL with time and memory costs that scale linearly in the parameter count [1]. The architecture integrates into existing streaming algorithms for both discrete and continuous control tasks [1]. On a MemoryChain diagnostic with chain lengths ranging from 2 to 128, the method sustained performance while streaming TBPTT(1) baselines using feedforward, GRU, and RTU networks collapsed [1]. On five POPGym tasks and on partially observable MuJoCo continuous-control benchmarks, the streaming approach proved competitive with batched Proximal Policy Optimization (PPO) on POPGym and recovered a substantial fraction of batched performance on masked MuJoCo, despite using no replay buffer or batched updates [1]. The results suggest that exact RTRL, long considered impractical for online learning, can be made tractable through architectural choices that constrain the recurrent weight matrix.