On the Relationship Between CoCoA and ADMM for Distributed Empirical Risk Minimization

Researchers have found a unified framework for understanding two influential families of algorithms used in distributed empirical risk minimization (ERM): CoCoA-type and ADMM-type algorithms.

A recent study on arXiv^[1] has shown that CoCoA-type algorithms, derived from distributed dual coordinate ascent, and ADMM-type algorithms, derived from consensus and proximal splitting, can be written in a common update form involving a global primal variable and block dual variables. This reformulation reveals previously hidden connections between the two types of algorithms. For instance, ridge-regularized CoCoA coincides with a particular proximal ADMM scheme at the level of the dual update^[1]. Moreover, consensus ADMM on the primal problem is equivalent to proximal ADMM on the dual problem under an explicit parameter mapping. The study also indicates that ADMM-type algorithms can perform at least as well as CoCoA under ridge-regularized ERM problems. Another related study on arXiv^[2] extended the Convex Gaussian Min-Max Theorem (CGMT) to non-Gaussian settings and found that the ERM estimator's mean and covariance can be approximated using an asymptotic min-max characterization. The projection of the ERM estimator onto a test covariate follows a convolution of a non-Gaussian distribution and a centered Gaussian variable^[2].