Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimization
Researchers have developed a new 2D parallel stochastic gradient descent method called HybridSGD to improve distributed-memory optimization algorithms for machine learning.
The HybridSGD method achieves better convergence than Federated Stochastic Gradient Descent with Averaging (FedAvg) at similar processor scales, according to a study published on arxiv.org[1]. The researchers implemented HybridSGD in C++ and MPI and evaluated its performance on a Cray EX supercomputing system. They found that HybridSGD attained speedups of up to 121 times over FedAvg[1]. The method generalizes prior work on 1D $s$-step SGD and 1D FedAvg, providing a continuous performance trade-off between the two baseline algorithms. Another study on arxiv.org[2] provided context on stochastic gradient descent, noting that it introduces noise in gradient estimation, which variance reduction techniques like SVRG and SAG aim to mitigate. The proposed framework interprets mini-batch gradients through survey sampling theory[2]. Empirical results from the second study showed performance gains in 71-86% of the experiments[2].
research-paperapplicationinfrastructuremodel-releasetool-release
Background sources we checked (4)
- arxiv.org ↗ Distributed-memory implementations of numerical optimization algorithm, such as stochastic gradient descent (SGD), require interprocessor communication at every iteration of the algorithm. On modern distributed-memory clusters where communication is more expensive than computatio…
- en.wikipedia.org ↗ In computer science and operations research, the ant colony optimization algorithm (ACO) is a probabilistic technique for solving computational problems that can be reduced to finding good paths through graphs. Artificial ants represent multi-agent methods inspired by the behavio…
- en.wikipedia.org ↗ A convolutional neural network (CNN) is a type of feedforward neural network that learns features via filter (or kernel) optimization. This type of deep learning network has been applied to process and make predictions from many different types of data including text, images and …
- en.wikipedia.org ↗ Adversarial machine learning is the study of the attacks on machine learning algorithms, and of the defenses against such attacks. Machine learning techniques are mostly designed to work on specific problem sets, under the assumption that the training and test data are generated …