How do you choose a loss function, and why cross-entropy for classification versus MSE for regression?

The loss function defines what the network minimizes, so it should match the task. For regression, mean squared error (MSE) penalizes the squared difference between prediction and target; it assumes Gaussian noise and is sensitive to outliers, where mean absolute error or Huber loss can be more robust. For classification, cross-entropy compares the predicted probability distribution to the true labels, paired with softmax for multi-class or sigmoid for binary. Cross-entropy is preferred over MSE for classification because it produces stronger, well-behaved gradients when predictions are confidently wrong, whereas MSE combined with sigmoid saturates and yields tiny gradients that slow learning. Cross-entropy also has a clean probabilistic interpretation as maximizing the likelihood of the correct labels. Choosing a loss aligned with the task and output activation is essential for stable, efficient training.