GShard
GShard
**GShard** is a intra-layer parallel distributed method. It consists of set of simple APIs for annotations, and a compiler extension in XLA for automatic parallelization.
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
https://paperswithcode.com/paper/gshard-scaling-giant-models-with-conditional
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding