Bellman-Taylor Score Decoding for Markov Decision Processes with State-Dependent Feasible Action Sets

A new framework called Bellman–Taylor score decoding aims to let standard deep reinforcement learning algorithms handle Markov decision processes where the set of allowable actions changes with each state, a common obstacle in operations research [1]. Many real-world sequential decision problems, from routing to inventory management, are modeled as Markov decision processes (MDPs) in which the actions available at any step depend on the current state and are defined implicitly by operational constraints [1][2]. Standard deep reinforcement learning (DRL) algorithms are built for either a fixed catalog of discrete actions or a simple Euclidean action space, making them difficult to apply directly to these constrained settings [2]. The Bellman–Taylor score decoding framework, submitted in 2026, addresses this mismatch by shifting policy learning into a Euclidean score space and pairing it with an action decoder that enforces feasibility [1][2]. The approach is motivated by a Taylor expansion of the optimal action-value function [2]. By mapping state-dependent feasible actions to scores in a Euclidean space, the method creates what the authors call a latent-score MDP. This transformed problem can be optimized by off-the-shelf DRL algorithms without requiring the decoder to be differentiable [1][2]. The researchers provide a performance guarantee that breaks the optimality gap into two components: a structural approximation error, which captures how well the score space represents the original problem, and an algorithmic learning error, which reflects the DRL agent’s training performance [1][2]. To test the framework, the team applied it to a queueing network control problem, where the policy learns a state-dependent index-based dispatching rule [1][2]. In numerical experiments, the method achieved near-optimal performance on small instances and showed considerable improvement over benchmark approaches on larger systems [1][2]. The results suggest that decoupling feasibility from learning can widen the set of constrained operations problems that DRL can tackle without custom algorithmic modifications. The work appears on arXiv under the artificial intelligence category [1].