Gradient Sparsification
Gradient Sparsification
**Gradient Sparsification** is a technique for distributed training that sparsifies stochastic gradients to reduce the communication cost, with minor increase in the number of iterations. The key idea behind our sparsification technique is to drop some coordinates of the stochastic gradient and appropriately amplify the remaining coordinates to ensure the unbiasedness of the sparsified stochastic gradient. The sparsification approach can significantly reduce the coding length of the stochastic gradient and only slightly increase the variance of the stochastic gradient.
Gradient Sparsification for Communication-Efficient Distributed Optimization
https://paperswithcode.com/paper/gradient-sparsification-for-communication
Gradient Sparsification for Communication-Efficient Distributed Optimization