Linear Warmup With Cosine Annealing
Linear Warmup With Cosine Annealing
**Linear Warmup With Cosine Annealing** is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule afterwards.