Mesh-TensorFlow
Mesh-TensorFlow
**Mesh-TensorFlow** is a language for specifying a general class of distributed tensor computations. Where data-parallelism can be viewed as splitting tensors and operations along the "batch" dimension, in Mesh-TensorFlow, the user can specify any tensor dimensions to be split across any dimensions of a multi-dimensional mesh of processors. A MeshTensorFlow graph compiles into a SPMD program consisting of parallel operations coupled with collective communication primitives such as Allreduce.
Mesh-TensorFlow: Deep Learning for Supercomputers
https://paperswithcode.com/paper/mesh-tensorflow-deep-learning-for
Mesh-TensorFlow: Deep Learning for Supercomputers