Why does weight initialization matter, and what schemes are commonly used?

Initialization sets the starting weights before training and strongly affects whether a deep network trains well. If all weights are identical (for example all zeros), every neuron in a layer computes the same thing and receives the same gradient, so they never differentiate; this is the symmetry problem, which is why we initialize randomly. If weights are too large, activations and gradients explode; too small, they vanish. Good schemes keep the variance of activations and gradients stable across layers. Xavier/Glorot initialization scales variance by the number of input and output units and suits tanh or sigmoid. He initialization scales by the number of inputs and is designed for ReLU, accounting for the fact that ReLU zeros out half the activations. Biases are usually initialized to zero. Proper initialization, combined with normalization, is key to training very deep networks.