Path-Coupled Bellman Flows for Distributional Reinforcement Learning

A new continuous-time method for distributional reinforcement learning, Path-Coupled Bellman Flows (PCBF), addresses boundary mismatch and high-variance bootstrapping that have limited prior flow-based approaches, according to research posted on arXiv [1]. The framework, introduced by Hao Yan, learns return distributions using flow matching with what the paper terms source-consistent Bellman-coupled paths. The current path starts from the required base prior at time zero, reaches the Bellman target at time one, and maintains a pathwise affine relation to the successor flow at intermediate times [1][3]. This design separates the geometric requirement of flow matching from the stochasticity of Bellman bootstrapping [3]. Earlier distributional RL methods often model returns as categorical distributions over discrete bins or estimate a finite number of quantiles, leaving questions about fine-grained structure unanswered [5]. More recent flow-based models have shown promise but can suffer from boundary mismatch at the flow source or from high-variance bootstrapping when current and successor noises are independent [2][4]. PCBF couples current and successor return flows through shared base noise, aligning their intermediate trajectories rather than enforcing Bellman consistency only at the endpoint [3]. The method uses a lambda-parameterized control-variate target: when lambda equals zero, it recovers an unbiased sample Bellman target, while lambda greater than zero trades controlled bias for variance reduction [1][4]. This pathwise structure induces a family of training targets that interpolates between direct sample-based Bellman supervision and variance-reduced supervision using successor-flow velocity predictions [3]. On the theoretical side, the paper characterizes the population-optimal velocity field induced by the PCBF path, analyzes the bias-variance behavior of the lambda-target, and shows that shared-noise Bellman generator updates inherit gamma-contraction and induce a t-gamma contraction for PCBF interpolants [3][4]. Experiments on analytically tractable Markov reward processes, OGBench, and D4RL Adroit demonstrate improved distributional fidelity and training stability, along with competitive offline RL performance [1][3]. A separate flow-matching framework called Value Flows, detailed in earlier work, formulated a distributional flow-matching objective that generates probability density paths satisfying the distributional Bellman equation and reported a 1.3-times average improvement in success rates across 37 state-based and 25 image-based benchmark tasks [5]. PCBF builds on this lineage by directly addressing the boundary-mismatch and variance issues through its coupled-path construction [2][3].