Stochastic Weight Averaging
Stochastic Weight Averaging
**Stochastic Weight Averaging** is an optimization procedure that averages multiple points along the trajectory of [SGD](https://paperswithcode.com/method/sgd), with a cyclical or constant learning rate. On the one hand it averages weights, but it also has the property that, with a cyclical or constant learning rate, SGD proposals are approximately sampling from the loss surface of the network, leading to stochastic weights and helping to discover broader optima.
Averaging Weights Leads to Wider Optima and Better Generalization
https://paperswithcode.com/paper/averaging-weights-leads-to-wider-optima-and
Averaging Weights Leads to Wider Optima and Better Generalization