On the Condition Number Dependency in Bilevel Optimization

Researchers have made progress in understanding the complexity of bilevel optimization, establishing a new lower bound for finding an epsilon-stationary point with first-order methods.

Bilevel optimization involves minimizing an objective function defined by an upper-level problem whose feasible region is the solution of a lower-level problem^[1]. Recent studies have focused on the oracle complexity of finding an epsilon-stationary point using first-order methods. A new lower bound of Omega(kappa_y^{5/2} epsilon^{-2}) has been established, where kappa_y is the lower-level condition number^[1]. This development builds on previous work that achieved a near-optimal upper bound in terms of epsilon, with a complexity of O(bar kappa_y^4 epsilon^{-2})^[1]. The optimal dependency on the condition number was previously unknown. The new lower bound has implications for various settings, including high-order smooth functions and stochastic oracles. For instance, researchers have shown lower bounds of Omega(kappa_y^{31/14} epsilon^{-12/7}) for second-order smooth problems and Omega(kappa_y^4 epsilon^{-4}) for smooth stochastic problems^[1]. Upper bound guarantees for bilevel optimization have been widely studied, but progress on lower bounds has been limited due to the complexity of the bilevel structure^[2]. The new findings highlight substantial gaps between current upper and lower bounds for bilevel optimization, underscoring the need for further research^[2].