Pipelined Backpropagation
Pipelined Backpropagation
**Pipelined Backpropagation** is an asynchronous pipeline parallel training algorithm. It was first introduced by Petrowski et al (1993). It avoids fill and drain overhead by updating the weights without draining the pipeline first. This results in weight inconsistency, the use of different weights on the forward and backward passes for a given micro-batch. The weights used to produce a particular gradient may also have been updated when the gradient is applied, resulting in stale (or delayed) gradients. For these reasons PB resembles Asynchronous [SGD](https://paperswithcode.com/method/sgd) and is not equivalent to standard SGD. Finegrained pipelining increases the number of pipeline stages and hence increases the weight inconsistency and delay.
Pipelined Backpropagation at Scale: Training Large Models without Batches
https://paperswithcode.com/paper/pipelined-backpropagation-at-scale-training
Pipelined Backpropagation at Scale: Training Large Models without Batches