The Value Function Semi-Algebraic Set in Partially Observable Markov Decision Processes

Researchers have made advancements in understanding policy gradient algorithms and proposing new frameworks for reinforcement learning, with recent submissions to arXiv.org on June 2-3, 2026.

A paper submitted on June 2 characterized the feasible set of value functions in infinite-horizon partially observable Markov decision processes (POMDPs) as a semi-algebraic set^[1]. The characterization is based on explicit polynomial inequalities determined by the transition dynamics, observation kernel, and reward structure of the POMDP. This result extends prior work for fully observable Markov decision processes to the partially observable setting. On the same day, another paper proposed SDPG, a self-distilled policy-gradient framework that improves stability and performance in reinforcement learning^[2]. SDPG combines group-relative verifier advantages with normalized standard deviation and exact full-vocabulary on-policy self-distillation, and includes reference-policy KL regularization. A subsequent submission on June 3 investigated policy gradient algorithms within a continuous-time RMDP framework^[3]. The paper derived policy gradients and adversarial gradients using pathwise and adjoint-based formulas for stochastic and ordinary differential equations, and proposed double-loop optimisers and mean-field optimisers with specific convergence rates.