Finite-Width Neural Tangent Kernels from Feynman Diagrams

Researchers have made advancements in neural tangent kernels (NTKs), a tool for analyzing deep neural networks, by developing methods to include finite-width effects and improve convergence in decentralized federated learning.

NTKs are a powerful tool for analyzing deep, non-linear neural networks, allowing for full analytic control over training dynamics at infinite width^[1]. However, finite-width effects are crucial for understanding certain training properties. Researchers have introduced Feynman diagrams to compute these effects, simplifying algebraic manipulations and enabling layer-wise recursion relations. They demonstrated that their framework can extend stability results from preactivations to NTKs and showed that first-order corrections for arbitrary inputs follow the statistics of sampled neural networks for widths $n\ngtr 20$^[1]. In a separate development, researchers found that decentralized federated learning (DFL) converges slowly under statistical heterogeneity, but NTK methods achieve faster convergence than gradient-based updates. A new method, SPARK, converges about 3x faster than baselines under high heterogeneity and lowers total communication to a target accuracy by up to about 70%^[2].