Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence

Researchers have proposed an alternative objective to minimize the forward Fisher divergence in Gaussian mixture models, leading to better optimization properties, and developed a new gradient descent method called piecewise polynomial interpolation-based gradient descent (PPI-GD)^[1][2].

The score matching problem is a central training objective in modern generative modeling and diffusion models. Minimizing the forward Fisher divergence, a standard approach, can lead to undesirable convergence behavior in Gaussian mixture model settings^[1]. Researchers have studied an alternative objective, the reverse Fisher divergence, which takes expectation with respect to the student distribution. Analysis of gradient descent (GD) for fitting Gaussian mixture models shows global convergence guarantees under certain conditions. A new inexact gradient method, PPI-GD, approximates the full gradient in each iteration by querying the first-order oracle at equidistant points in the data domain and constructs polynomial interpolants of the resulting gradient samples. PPI-GD outperforms several GD variants when the loss function is sufficiently smooth^[2].