ZeRO
ZeRO
**Zero Redundancy Optimizer (ZeRO)** is a sharded data parallel method for distributed training. ZeRODP removes the memory state redundancies across data-parallel processes by partitioning the model states instead of replicating them, and it retains the compute/communication efficiency by retaining the computational granularity and communication volume of DP using a dynamic communication schedule during training.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
https://paperswithcode.com/paper/zero-memory-optimization-towards-training-a
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models