What is layer-wise learning rate decay, and why is it beneficial for training transformers?

Layer-wise learning rate decay is a training strategy that applies different learning rates to different layers of a neural network, typically with lower rates for lower layers and higher rates for upper layers. This approach is beneficial for transformers because it allows the model to learn more complex features in deeper layers while stabilizing the training of earlier layers. It helps prevent overfitting in lower layers that are already well-trained and encourages faster convergence overall.