AutoSync
AutoSync
**AutoSync** is a pipeline for automatically optimizing synchronization strategies, given model structures and resource specifications, in data-parallel distributed machine learning. By factorizing the synchronization strategy with respect to each trainable building block of a DL model, we can construct a valid and large strategy space spanned by multiple factors. AutoSync efficiently navigates the space and locates the optimal strategy. AutoSync leverages domain knowledge about synchronization systems to reduce the search space, and is equipped with a domain adaptive simulator, which combines principled communication modeling and data-driven ML models, to estimate the runtime of strategy proposals without launching real distributed execution.
AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning
https://paperswithcode.com/paper/autosync-learning-to-synchronize-for-data
AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning