Accordion
Accordion
**Accordion** is a gradient communication scheduling algorithm that is generic across models while imposing low computational overheads. Accordion inspects the change in the gradient norms to detect critical regimes and adjusts the communication schedule dynamically. Accordion works for both adjusting the gradient compression rate or the batch size without additional parameter tuning.
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
https://paperswithcode.com/paper/accordion-adaptive-gradient-communication-via
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification