p-PSO: A Penalized Particle Swarm Optimization Technique for Finding D-Optimal Designs with Mixed Factors in Generalized Linear Models

Researchers have proposed two new optimization methods for generalized linear models, a penalized Particle Swarm Optimization (PSO) approach and a mixed-precision communication-avoiding stochastic gradient descent (SGD) method.

A new penalized PSO approach, named p-PSO, has been proposed for finding D-optimal designs with mixed factors in generalized linear models. The method introduces a penalty formulation for constrained optimization, enabling the direct use of an off-the-shelf PSO algorithm^[1]. Meanwhile, a mixed-precision communication-avoiding SGD method has been proposed for generalized linear models on GPUs, achieving a speedup over traditional SGD. This method amortizes communication over multiple iterations and leverages modern GPUs to accelerate computations, resulting in a 5.1--6.8x speedup over FP32 SGD on certain problems^[2]. The mixed-precision CA-SGD method also matches FP32 SGD loss within 0.5% on logistic, linear, and Poisson problems. The development of these methods addresses the challenges of optimizing generalized linear models, where the Fisher information matrix depends on unknown parameters and classical algorithms have offered limited solutions.