Global Policy-Space Response Oracles for Two-Player Zero-Sum Games

A new algorithm called Global PSRO can approximate Nash equilibria in two-player zero-sum games using fewer policy iterations than prior methods, according to a paper submitted in 2026 [1]. The approach directly evaluates population quality during expansion to reduce exploitability [2]. The Policy-Space Response Oracles (PSRO) framework scales equilibrium computation to large zero-sum games by iteratively expanding a restricted strategy set using deep reinforcement learning (DRL) [1]. A central challenge is constructing, under limited computational budgets, a small strategy population whose induced game closely approximates the full game [2]. Existing PSRO variants typically expand the population using best responses to meta-strategies computed from restricted-game payoffs, which can lead to inefficient expansions that provide limited global improvement [1]. To address this, the researchers propose guiding population expansion by directly evaluating the post-expansion population quality [2]. They adopt Population Exploitability (PE) to measure how well a restricted strategy set represents the full game, and introduce a two-phase exploration–selection framework that explicitly minimizes PE during expansion [1]. This framework is instantiated as Global PSRO, a practical DRL-based algorithm that efficiently generates candidate responses and estimates PE via parameter-sharing conditional neural networks [2]. Experiments across multiple two-player zero-sum games show that Global PSRO achieves lower exploitability and approximates Nash equilibria with significantly fewer policy iterations than prior PSRO methods [1]. The work was submitted to arXiv on 27 May 2026 under the Computer Science and Artificial Intelligence category [1]. Deep reinforcement learning, the technique underpinning Global PSRO, has been a focus of major technology corporations. Google, for instance, has invested heavily in AI through its DeepMind subsidiary and offers machine learning APIs such as TensorFlow [5]. The company has also developed custom AI chips, known as TPUs, to accelerate such workloads [5]. While the Global PSRO paper does not disclose specific hardware used, the computational demands of DRL-based equilibrium computation often require specialized infrastructure of the kind provided by cloud computing platforms [5].